REVIEW 3 major objections 5 minor 17 references
CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A combined subtask model with attention pooling and multi-objective training beats the baseline for verifying corporate ESG promises.
desk verdict A real external leaderboard score, but the abstract credits components that the submitted hybrid system doesn't isolate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the combined subtask model: a DeBERTa-v3-large encoder with two task-specific classifier heads sharing all transformer layers. Three additions do the work. First, attention pooling replaces the [CLS] token by computing $\alpha_i = \operatorname{softmax}(W_{\text{attn}} h_i)$ over hidden states $h_i$ and forming $r = \sum_{i=1}^{n} \alpha_i h_i$. Second, the raw text is enriched as $x_{\text{enriched}} = \text{[PAGE}_p\text{] [ESG REPORT]} + x_{\text{raw}}$, prepending a page marker and report-type tag. Third, the training objective is a weighted sum $L = 0.6\,L_{\text{promise}} + 0.4\,L_{\text{evidence}}$, with focal loss, gradient accumulation, and test-time augmentation over three forward passes whose probabilities are averaged and thresholded at 0.5.
What would settle it
Re-run both the combined model and the baseline with the same backbone, the same train/validation split, and multiple random seeds, then compare their scores on the private test set; if the combined model's edge is no larger than the standard deviation across seeds, the claimed improvement is not distinguishable from noise. A sharper test is to ablate attention pooling, metadata prepending, and test-time augmentation one at a time with everything else held fixed.
Extended reading notes
Core claim
The central claim is that promise verification in ESG reports improves when promise identification and supporting-evidence assessment are trained jointly on a shared transformer, with attention-based pooling and metadata-enriched inputs, rather than as separate classification tasks. The paper reports private leaderboard scores of 0.4994 for the ESG-BERT base model, 0.5094 for the feature-enhanced model, and 0.5268 for the combined subtask model, against a Kaggle baseline of 0.5227. Its final system therefore uses the combined model for subtasks 1 and 2 and the feature-enhanced model for clarity and timing subtasks. The paper also finds that prepended linguistic features (promise words, sentiment, metric counts, vague/specific terms, date spans) improve the ESG-BERT models only slightly, which it attributes to redundancy with domain-specific pre-training and to the small 400-instance training set.
Load-bearing premise
The paper's central claim rests on the assumption that the combined model's 0.0041-point lead over the baseline on the private leaderboard comes from its design choices, rather than from the different backbone, the different 90-10 data split, the multi-objective loss, the shorter sequence length, or run-to-run variation.
Editorial extensions
If this is right
- If the gain is real, joint training on promise identification and evidence assessment transfers knowledge between the two subtasks, since the combined model beats two separately trained ESG-BERT models.
- Attention pooling over token hidden states is a viable alternative to the [CLS] representation on this small imbalanced dataset.
- Prepending document metadata (page number, report type) can help a transformer locate promise-relevant context, a cheap addition that requires no extra labels.
- Test-time augmentation with probability averaging and a calibrated threshold is a practical variance-reduction tool for a 400-instance training set.
- The final recipe is modular: use the strongest model per subtask group and assemble predictions, which is how the paper obtains its leaderboard score.
Reading between the lines
- Because the combined model differs from the baseline in several ways at once (backbone, data split, loss, sequence length, training schedule), the 0.0041 gain should be read as an association, not a proven effect of any single component; a controlled ablation is the natural sequel.
- The same architecture could be extended to all four subtasks in one multi-objective model, since the paper itself notes that negative transfer between promise and evidence detection may have limited the gain; whether adding clarity and timing helps or hurts is an open question.
- The metadata markers are a cheap testable variable: replacing the real page number with a shuffled or constant tag would show whether the model exploits document position or merely a constant prefix.
- The result is specific to the English split; applying the recipe to the multilingual part of ML-Promise would test whether the hand-crafted linguistic tags and document-type markers transfer across languages.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports three model architectures for SemEval-2025 Task 6 (PromiseEval) on the English portion of the ML-Promise dataset: (i) an ESG-BERT base model with per-subtask classifier heads; (ii) an ESG-BERT model with hand-crafted linguistic feature tags prepended to the input; and (iii) a DeBERTa-v3-large multitask model for Subtasks 1 and 2 using attention pooling, document metadata tokens, a 0.6/0.4 multi-objective loss, and test-time augmentation. The authors report private leaderboard scores of 0.4994, 0.5094, and 0.5268 for these models, with the best score surpassing the provided Kaggle baseline of 0.5227. The abstract credits linguistic feature extraction, attention pooling, and multi-objective learning for the improvement.
Significance. The paper's main strength is that the evaluation uses an external held-out leaderboard, so the reported numbers are not fitted to the test set, and the authors are candid about the limitations of their study, including the absence of a systematic ablation. If the attribution were backed by controlled comparisons, the result would be a useful data point for promise-verification system design. As reported, however, the central attribution claim is not established: the best score comes from a hybrid submission rather than from the combined model alone, and multiple architectural and training choices are confounded. The paper is a reasonable system-description contribution, but it requires a careful revision to correct the attribution and to provide per-subtask or ablation evidence.
major comments (3)
- [Abstract; §6] The abstract states that 'our combined subtask approach achieving a leaderboard score of 0.5268', but Section 6 explicitly says that the Combined Model only worked on Tasks 1 and 2 and that the final submission used Model 3 for those tasks and the Feature-Enhanced Model for Tasks 3 and 4. The 0.5268 private score therefore belongs to a hybrid system, and the comparison with the 0.5227 baseline cannot be used to support claims about the combined model's components in isolation.
- [§6] The sentence 'We attribute this improvement to three factors: (1) multitask learning benefits from shared representations... (2) attention pooling... (3) test-time augmentation' is an attribution, not a demonstrated effect. Model 3 differs from Models 1 and 2 and from the baseline in the backbone (DeBERTa-v3-large vs ESG-BERT), the train/validation split (90/10 in §5.2 vs 4-fold 75/25 in §5.1), the sequence length (256 tokens), the loss weighting (Eq. 5), gradient accumulation, cosine schedule, and threshold calibration. No ablation isolates any of the named factors, and Section 7 defers a systematic ablation study to future work, which acknowledges the missing support.
- [Table 1; §5] The paper reports only aggregate leaderboard scores and never reports per-subtask results or the official metric definition. Since the submitted system is a hybrid across subtasks, per-subtask breakdowns are necessary to determine whether the gain over the baseline comes from Model 3 on Tasks 1 and 2, from Model 2 on Tasks 3 and 4, or from run-to-run noise. The 0.0041 absolute difference between 0.5268 and 0.5227 is small, and no repeated runs, confidence intervals, or significance tests are provided.
minor comments (5)
- [Eq. (2)] Equation (2) is typeset incorrectly; it should read r = \sum_{i=1}^{n} \alpha_i h_i, with an explicit summation symbol.
- [§6] The phrase '1.74% absolute improvement over the baseline' is inconsistent with Table 1: 0.5268 - 0.5227 = 0.0041, which is 0.41 percentage points, not 1.74 percentage points.
- [§1, Table 1] The 'Kaggle Baseline' is not described; the paper should state what this baseline is and how its predictions were obtained.
- [Footnote 1] The GitHub URL in Footnote 1 is cut off after 'SemEval-2025-Task6'; please provide a complete, working URL so that the claimed code release is verifiable.
- [§4.3] The attention pooling parameters W_attn in Eq. (1) are not described; specifying their initialization and whether the attention layer is shared between the two task heads would improve reproducibility.
Circularity Check
No circularity: the leaderboard score is externally evaluated and the hybrid composition is a transparency issue, not a fitted or definitional reduction.
full rationale
The paper's central numerical claim is a SemEval private-leaderboard score (0.5268), which is computed by the task organizers on held-out test data. No quantity in the derivation chain is fitted to that test set, so the result is not equivalent to an input by construction. The only serious weakness is that the reported 'Combined Subtask Model' score is actually a hybrid: Section 6 states that Task 1 and 2 predictions come from the Combined Model while Tasks 3 and 4 come from the Feature-Enhanced Model. That makes the abstract attribution of the gain to attention pooling and multi-objective learning unsupported, but it is an experimental-confound/attribution issue, not circularity. The equations in Section 4.3 define a model architecture; they do not encode the leaderboard outcome. No load-bearing claim is justified by a self-citation: the references to ESG-BERT, DeBERTa, and prior work are external sources, and no 'uniqueness theorem' is invoked. The paper explicitly defers ablation to future work, which further confirms that the component attributions are hypotheses rather than fitted values. Since the evaluation is external and the architecture definitions are independent of the reported score, there is no circularity to flag.
Assumptions & free parameters
free parameters (3)
- Multi-objective loss weights =
0.6 (promise) / 0.4 (evidence)
- TTA threshold =
0.5
- Hand-crafted linguistic feature lists =
not enumerated in the paper
assumptions (4)
- domain assumption Pretrained transformer weights (ESG-BERT, DeBERTa-v3-large) transfer to the four PromiseEval subtasks.
- domain assumption Prepending feature tags and metadata markers is a usable input representation.
- domain assumption Single-run leaderboard scores are stable estimates of system quality.
- standard math Standard deep learning components (softmax attention pooling, GELU, layer norm, focal loss) behave as expected.
invented entities (1)
-
[PAGEp] and [ESG REPORT] metadata tokens
Cite this review
Pith. "Pith review of CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification." pith.science (2026). https://pith.science/paper/AIBK72EW
@misc{pith2026250523538,
author = {Pith},
title = {Pith review of: CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verification},
year = {2026},
howpublished = {\url{https://pith.science/paper/AIBK72EW}},
note = {Machine review of arXiv:2505.23538}
}
read the original abstract
This paper presents our approach to the SemEval-2025 Task~6 (PromiseEval), which focuses on verifying promises in corporate ESG (Environmental, Social, and Governance) reports. We explore three model architectures to address the four subtasks of promise identification, supporting evidence assessment, clarity evaluation, and verification timing. Our first model utilizes ESG-BERT with task-specific classifier heads, while our second model enhances this architecture with linguistic features tailored for each subtask. Our third approach implements a combined subtask model with attention-based sequence pooling, transformer representations augmented with document metadata, and multi-objective learning. Experiments on the English portion of the ML-Promise dataset demonstrate progressive improvement across our models, with our combined subtask approach achieving a leaderboard score of 0.5268, outperforming the provided baseline of 0.5227. Our work highlights the effectiveness of linguistic feature extraction, attention pooling, and multi-objective learning in promise verification tasks, despite challenges posed by class imbalance and limited training data.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. https://doi.org/10.18653/v1/N19-4010 FLAIR : An easy-to-use framework for state-of-the-art NLP . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics (Demonstrations) (NAACL-2019) , pages 54-...
-
[4]
Felix Armbrust, Henry Sch \"a fer, and Roman Klinger. 2020. https://aclanthology.org/2020.fnp-1.31/ A computational analysis of financial and environmental narratives within financial reports and its value for investors . In Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation, pages 181--194, Barce...
work page 2020
-
[5]
Julia Bingler, Mathias Kraus, and Markus Leippold. 2021. https://doi.org/10.2139/ssrn.3796152 Cheap talk and cherry-picking: What climatebert has to say on corporate climate risk disclosures . SSRN Electronic Journal
-
[6]
Chung-Chi Chen, Yohei Seki, Hakusen Shu, Ana \" s Lhuissier, Juyeon Kang, Hanwool Lee, Min-Yuh Day, and Hiroya Takamura. 2025. SemEval-2025 Task 6: Multinational, Multilingual, Multi-Industry Promise Verification . In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna
work page 2025
-
[7]
Shijie Chen, Yu Zhang, and Qiang Yang. 2024. http://arxiv.org/abs/2109.09138 Multi-Task Learning in Natural Language Processing: An Overview . ArXiv preprint, https://arxiv.org/abs/2109.09138
work page Pith review arXiv 2024
-
[8]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. http://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . ArXiv preprint, https://arxiv.org/abs/2006.03654
arXiv 2021
Show all 17 references
-
[9]
Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023. https://doi.org/10.18653/v1/2023.eacl-main.38 A survey of methods for addressing class imbalance in deep-learning based natural language processing . In Proceedings of the 17th Conference of the E...
2023 doi
-
[10]
Justin Johnson and Taghi Khoshgoftaar. 2019. https://doi.org/10.1186/s40537-019-0192-5 Survey on deep learning with class imbalance . Journal of Big Data, 6:27
2019 doi
-
[11]
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. http://arxiv.org/abs/1901.11504 Multi-task deep neural networks for natural language understanding . ArXiv preprint,https://arxiv.org/abs/1901.11504
2019 arXiv
-
[12]
Mukut Mukherjee and Sree Charan Pothireddi. 2021. ESG-BERT: Domain Specific BERT Model for Text Mining in Sustainable Investing . https://huggingface.co/nbroad/ESG-BERT. Accessed: February 2025
2021
-
[13]
Hamilton, Dan McFarland, and Dan Jurafsky
Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016. https://doi.org/10.18653/v1/P16-1111 Predicting the rise and fall of scientific topics from trends in their rhetorical framing . In Proceedings of the 54th Annual Meeting of the Association for...
2016 doi
-
[14]
Yohei Seki, Hakusen Shu, Anaïs Lhuissier, Hanwool Lee, Juyeon Kang, Min-Yuh Day, and Chung-Chi Chen. 2024. http://arxiv.org/abs/2411.04473 ML-Promise: A Multilingual Dataset for Corporate Promise Verification . arXiv preprint arXiv:2411.04473
2024 arXiv
-
[15]
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. 2021. http://arxiv.org/abs/2011.11156 Better aggregation in test-time augmentation . ArXiv preprint, https://arxiv.org/abs/2011.11156
2021 arXiv
-
[16]
Shuo Yang, Zhiqiang Zhang, Jun Zhou, Yang Wang, Wang Sun, Xingyu Zhong, Yanming Fang, Quan Yu, and Yuan Qi. 2021. Financial risk analysis for smes with graph-based supply chain mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence...
2021
-
[17]
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. https://doi.org/10.18653/v1/N16-1174 Hierarchical attention networks for document classification . In Proceedings of the 2016 Conference of the North American Chapter of the Association for Com...
2016 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.