Pith. sign in

REVIEW 3 major objections 5 minor 17 references

CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A combined subtask model with attention pooling and multi-objective training beats the baseline for verifying corporate ESG promises.

desk verdict A real external leaderboard score, but the abstract credits components that the submitted hybrid system doesn't isolate. read the letter →

arxiv 2505.23538 v1 pith:AIBK72EW submitted 2025-05-29 cs.CL cs.AI

classification cs.CLcs.AI
keywords promiseverificationESGreportsmultitasklearningattentionpoolinglinguisticfeaturestest-timeaugmentationclassimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a modest but consistent improvement in automatically verifying whether corporate ESG statements really contain promises and evidence. Working from the English portion of the ML-Promise dataset, it compares three architectures: a frozen ESG-BERT with task-specific heads, that model augmented with hand-crafted linguistic tags, and a combined subtask model that shares a DeBERTa-v3-large encoder between promise identification and evidence assessment. The combined model reaches 0.5268 on the private test leaderboard of SemEval-2025 Task 6, above the provided baseline of 0.5227 and above the paper's own feature-enhanced model at 0.5094. The paper attributes the gain to attention pooling, document-metadata prepending, multi-objective learning, and test-time augmentation, while noting the gain is small relative to the added complexity.

What carries the argument

The carrying mechanism is the combined subtask model: a DeBERTa-v3-large encoder with two task-specific classifier heads sharing all transformer layers. Three additions do the work. First, attention pooling replaces the [CLS] token by computing $\alpha_i = \operatorname{softmax}(W_{\text{attn}} h_i)$ over hidden states $h_i$ and forming $r = \sum_{i=1}^{n} \alpha_i h_i$. Second, the raw text is enriched as $x_{\text{enriched}} = \text{[PAGE}_p\text{] [ESG REPORT]} + x_{\text{raw}}$, prepending a page marker and report-type tag. Third, the training objective is a weighted sum $L = 0.6\,L_{\text{promise}} + 0.4\,L_{\text{evidence}}$, with focal loss, gradient accumulation, and test-time augmentation over three forward passes whose probabilities are averaged and thresholded at 0.5.

What would settle it

Re-run both the combined model and the baseline with the same backbone, the same train/validation split, and multiple random seeds, then compare their scores on the private test set; if the combined model's edge is no larger than the standard deviation across seeds, the claimed improvement is not distinguishable from noise. A sharper test is to ablate attention pooling, metadata prepending, and test-time augmentation one at a time with everything else held fixed.

Watch

Extended reading notes

Core claim

The central claim is that promise verification in ESG reports improves when promise identification and supporting-evidence assessment are trained jointly on a shared transformer, with attention-based pooling and metadata-enriched inputs, rather than as separate classification tasks. The paper reports private leaderboard scores of 0.4994 for the ESG-BERT base model, 0.5094 for the feature-enhanced model, and 0.5268 for the combined subtask model, against a Kaggle baseline of 0.5227. Its final system therefore uses the combined model for subtasks 1 and 2 and the feature-enhanced model for clarity and timing subtasks. The paper also finds that prepended linguistic features (promise words, sentiment, metric counts, vague/specific terms, date spans) improve the ESG-BERT models only slightly, which it attributes to redundancy with domain-specific pre-training and to the small 400-instance training set.

Load-bearing premise

The paper's central claim rests on the assumption that the combined model's 0.0041-point lead over the baseline on the private leaderboard comes from its design choices, rather than from the different backbone, the different 90-10 data split, the multi-objective loss, the shorter sequence length, or run-to-run variation.

Editorial extensions

If this is right

  • If the gain is real, joint training on promise identification and evidence assessment transfers knowledge between the two subtasks, since the combined model beats two separately trained ESG-BERT models.
  • Attention pooling over token hidden states is a viable alternative to the [CLS] representation on this small imbalanced dataset.
  • Prepending document metadata (page number, report type) can help a transformer locate promise-relevant context, a cheap addition that requires no extra labels.
  • Test-time augmentation with probability averaging and a calibrated threshold is a practical variance-reduction tool for a 400-instance training set.
  • The final recipe is modular: use the strongest model per subtask group and assemble predictions, which is how the paper obtains its leaderboard score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the combined model differs from the baseline in several ways at once (backbone, data split, loss, sequence length, training schedule), the 0.0041 gain should be read as an association, not a proven effect of any single component; a controlled ablation is the natural sequel.
  • The same architecture could be extended to all four subtasks in one multi-objective model, since the paper itself notes that negative transfer between promise and evidence detection may have limited the gain; whether adding clarity and timing helps or hurts is an open question.
  • The metadata markers are a cheap testable variable: replacing the real page number with a shuffled or constant tag would show whether the model exploits document position or merely a constant prefix.
  • The result is specific to the English split; applying the recipe to the multilingual part of ML-Promise would test whether the hand-crafted linguistic tags and document-type markers transfer across languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript reports three model architectures for SemEval-2025 Task 6 (PromiseEval) on the English portion of the ML-Promise dataset: (i) an ESG-BERT base model with per-subtask classifier heads; (ii) an ESG-BERT model with hand-crafted linguistic feature tags prepended to the input; and (iii) a DeBERTa-v3-large multitask model for Subtasks 1 and 2 using attention pooling, document metadata tokens, a 0.6/0.4 multi-objective loss, and test-time augmentation. The authors report private leaderboard scores of 0.4994, 0.5094, and 0.5268 for these models, with the best score surpassing the provided Kaggle baseline of 0.5227. The abstract credits linguistic feature extraction, attention pooling, and multi-objective learning for the improvement.

Significance. The paper's main strength is that the evaluation uses an external held-out leaderboard, so the reported numbers are not fitted to the test set, and the authors are candid about the limitations of their study, including the absence of a systematic ablation. If the attribution were backed by controlled comparisons, the result would be a useful data point for promise-verification system design. As reported, however, the central attribution claim is not established: the best score comes from a hybrid submission rather than from the combined model alone, and multiple architectural and training choices are confounded. The paper is a reasonable system-description contribution, but it requires a careful revision to correct the attribution and to provide per-subtask or ablation evidence.

major comments (3)
  1. [Abstract; §6] The abstract states that 'our combined subtask approach achieving a leaderboard score of 0.5268', but Section 6 explicitly says that the Combined Model only worked on Tasks 1 and 2 and that the final submission used Model 3 for those tasks and the Feature-Enhanced Model for Tasks 3 and 4. The 0.5268 private score therefore belongs to a hybrid system, and the comparison with the 0.5227 baseline cannot be used to support claims about the combined model's components in isolation.
  2. [§6] The sentence 'We attribute this improvement to three factors: (1) multitask learning benefits from shared representations... (2) attention pooling... (3) test-time augmentation' is an attribution, not a demonstrated effect. Model 3 differs from Models 1 and 2 and from the baseline in the backbone (DeBERTa-v3-large vs ESG-BERT), the train/validation split (90/10 in §5.2 vs 4-fold 75/25 in §5.1), the sequence length (256 tokens), the loss weighting (Eq. 5), gradient accumulation, cosine schedule, and threshold calibration. No ablation isolates any of the named factors, and Section 7 defers a systematic ablation study to future work, which acknowledges the missing support.
  3. [Table 1; §5] The paper reports only aggregate leaderboard scores and never reports per-subtask results or the official metric definition. Since the submitted system is a hybrid across subtasks, per-subtask breakdowns are necessary to determine whether the gain over the baseline comes from Model 3 on Tasks 1 and 2, from Model 2 on Tasks 3 and 4, or from run-to-run noise. The 0.0041 absolute difference between 0.5268 and 0.5227 is small, and no repeated runs, confidence intervals, or significance tests are provided.
minor comments (5)
  1. [Eq. (2)] Equation (2) is typeset incorrectly; it should read r = \sum_{i=1}^{n} \alpha_i h_i, with an explicit summation symbol.
  2. [§6] The phrase '1.74% absolute improvement over the baseline' is inconsistent with Table 1: 0.5268 - 0.5227 = 0.0041, which is 0.41 percentage points, not 1.74 percentage points.
  3. [§1, Table 1] The 'Kaggle Baseline' is not described; the paper should state what this baseline is and how its predictions were obtained.
  4. [Footnote 1] The GitHub URL in Footnote 1 is cut off after 'SemEval-2025-Task6'; please provide a complete, working URL so that the claimed code release is verifiable.
  5. [§4.3] The attention pooling parameters W_attn in Eq. (1) are not described; specifying their initialization and whether the attention layer is shared between the two task heads would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the leaderboard score is externally evaluated and the hybrid composition is a transparency issue, not a fitted or definitional reduction.

full rationale

The paper's central numerical claim is a SemEval private-leaderboard score (0.5268), which is computed by the task organizers on held-out test data. No quantity in the derivation chain is fitted to that test set, so the result is not equivalent to an input by construction. The only serious weakness is that the reported 'Combined Subtask Model' score is actually a hybrid: Section 6 states that Task 1 and 2 predictions come from the Combined Model while Tasks 3 and 4 come from the Feature-Enhanced Model. That makes the abstract attribution of the gain to attention pooling and multi-objective learning unsupported, but it is an experimental-confound/attribution issue, not circularity. The equations in Section 4.3 define a model architecture; they do not encode the leaderboard outcome. No load-bearing claim is justified by a self-citation: the references to ESG-BERT, DeBERTa, and prior work are external sources, and no 'uniqueness theorem' is invoked. The paper explicitly defers ablation to future work, which further confirms that the component attributions are hypotheses rather than fitted values. Since the evaluation is external and the architecture definitions are independent of the reported score, there is no circularity to flag.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The ledger is light: the paper introduces no new physical entities and its evaluation is external, so circularity burden is low. The main hand-chosen elements are the 0.6/0.4 loss weights, the 0.5 TTA threshold, the language-model backbones, and the unpublished feature-term lists, none of which are ablated.

free parameters (3)
  • Multi-objective loss weights = 0.6 (promise) / 0.4 (evidence)
    Eq. 5 fixes L = 0.6 L_promise + 0.4 L_evidence. The weights are chosen by hand with no sensitivity analysis, and the claim that multi-objective learning helps depends on them.
  • TTA threshold = 0.5
    Section 5.2 says predictions use 'calibrated thresholds (0.5)', a hand-chosen decision boundary affecting the final binary outputs.
  • Hand-crafted linguistic feature lists = not enumerated in the paper
    Section 4.2 relies on lists of promise terms, proof terms, vague terms, specific terms, and timeframe terms; these are chosen by the authors, not published in the paper, and are not given independent justification.
assumptions (4)
  • domain assumption Pretrained transformer weights (ESG-BERT, DeBERTa-v3-large) transfer to the four PromiseEval subtasks.
    Sections 4.1 and 4.3 fine-tune only the last layers of pretrained models; if the pretraining distribution does not cover ESG promise language, the systems lose their main capacity.
  • domain assumption Prepending feature tags and metadata markers is a usable input representation.
    Section 4.2 and Eq. 4 modify input text with tags ('POSITIVE Sentiment', '[PAGEp] [ESG REPORT]'); the paper itself concedes this may create a 'structural disconnect' (Section 6).
  • domain assumption Single-run leaderboard scores are stable estimates of system quality.
    Section 6 and Table 1 report one public and one private score per system with no variance or significance test; the headline comparison is a 0.0041 difference.
  • standard math Standard deep learning components (softmax attention pooling, GELU, layer norm, focal loss) behave as expected.
    Equations (1)-(3) and the focal loss mention in Section 5.2 rely on textbook definitions, treated as background.
invented entities (1)
  • [PAGEp] and [ESG REPORT] metadata tokens
    purpose: Context enrichment: Eq. 4 prepends page number and report-type markers so the transformer can condition on document metadata.
    Ad hoc input tokens introduced by the paper with no ablation showing they help; the paper reports no isolated test of this component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification." pith.science (2026). https://pith.science/paper/AIBK72EW

@misc{pith2026250523538,
  author       = {Pith},
  title        = {Pith review of: CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIBK72EW}},
  note         = {Machine review of arXiv:2505.23538}
}
read the original abstract

This paper presents our approach to the SemEval-2025 Task~6 (PromiseEval), which focuses on verifying promises in corporate ESG (Environmental, Social, and Governance) reports. We explore three model architectures to address the four subtasks of promise identification, supporting evidence assessment, clarity evaluation, and verification timing. Our first model utilizes ESG-BERT with task-specific classifier heads, while our second model enhances this architecture with linguistic features tailored for each subtask. Our third approach implements a combined subtask model with attention-based sequence pooling, transformer representations augmented with document metadata, and multi-objective learning. Experiments on the English portion of the ML-Promise dataset demonstrate progressive improvement across our models, with our combined subtask approach achieving a leaderboard score of 0.5268, outperforming the provided baseline of 0.5227. Our work highlights the effectiveness of linguistic feature extraction, attention pooling, and multi-objective learning in promise verification tasks, despite challenges posed by class imbalance and limited training data.

Figures

Figures reproduced from arXiv: 2505.23538 by the authors.

Figure 1
Figure 1. Class distribution across four subtasks in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. System architecture of the three Promise Verification models. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. https://doi.org/10.18653/v1/N19-4010 FLAIR : An easy-to-use framework for state-of-the-art NLP . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics (Demonstrations) (NAACL-2019) , pages 54-...

  4. [4]

    Felix Armbrust, Henry Sch \"a fer, and Roman Klinger. 2020. https://aclanthology.org/2020.fnp-1.31/ A computational analysis of financial and environmental narratives within financial reports and its value for investors . In Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation, pages 181--194, Barce...

  5. [5]

    Julia Bingler, Mathias Kraus, and Markus Leippold. 2021. https://doi.org/10.2139/ssrn.3796152 Cheap talk and cherry-picking: What climatebert has to say on corporate climate risk disclosures . SSRN Electronic Journal

  6. [6]

    Chung-Chi Chen, Yohei Seki, Hakusen Shu, Ana \" s Lhuissier, Juyeon Kang, Hanwool Lee, Min-Yuh Day, and Hiroya Takamura. 2025. SemEval-2025 Task 6: Multinational, Multilingual, Multi-Industry Promise Verification . In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna

  7. [7]

    Shijie Chen, Yu Zhang, and Qiang Yang. 2024. http://arxiv.org/abs/2109.09138 Multi-Task Learning in Natural Language Processing: An Overview . ArXiv preprint, https://arxiv.org/abs/2109.09138

  8. [8]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. http://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . ArXiv preprint, https://arxiv.org/abs/2006.03654

Show all 17 references
  1. [9]

    Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023. https://doi.org/10.18653/v1/2023.eacl-main.38 A survey of methods for addressing class imbalance in deep-learning based natural language processing . In Proceedings of the 17th Conference of the E...

  2. [10]

    Justin Johnson and Taghi Khoshgoftaar. 2019. https://doi.org/10.1186/s40537-019-0192-5 Survey on deep learning with class imbalance . Journal of Big Data, 6:27

  3. [11]

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. http://arxiv.org/abs/1901.11504 Multi-task deep neural networks for natural language understanding . ArXiv preprint,https://arxiv.org/abs/1901.11504

  4. [12]

    Mukut Mukherjee and Sree Charan Pothireddi. 2021. ESG-BERT: Domain Specific BERT Model for Text Mining in Sustainable Investing . https://huggingface.co/nbroad/ESG-BERT. Accessed: February 2025

  5. [13]

    Hamilton, Dan McFarland, and Dan Jurafsky

    Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016. https://doi.org/10.18653/v1/P16-1111 Predicting the rise and fall of scientific topics from trends in their rhetorical framing . In Proceedings of the 54th Annual Meeting of the Association for...

  6. [14]

    Yohei Seki, Hakusen Shu, Anaïs Lhuissier, Hanwool Lee, Juyeon Kang, Min-Yuh Day, and Chung-Chi Chen. 2024. http://arxiv.org/abs/2411.04473 ML-Promise: A Multilingual Dataset for Corporate Promise Verification . arXiv preprint arXiv:2411.04473

  7. [15]

    Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. 2021. http://arxiv.org/abs/2011.11156 Better aggregation in test-time augmentation . ArXiv preprint, https://arxiv.org/abs/2011.11156

  8. [16]

    Shuo Yang, Zhiqiang Zhang, Jun Zhou, Yang Wang, Wang Sun, Xingyu Zhong, Yanming Fang, Quan Yu, and Yuan Qi. 2021. Financial risk analysis for smes with graph-based supply chain mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence...

  9. [17]

    Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. https://doi.org/10.18653/v1/N16-1174 Hierarchical attention networks for document classification . In Proceedings of the 2016 Conference of the North American Chapter of the Association for Com...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.