Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

Gaurav Negi; John McCrae; MA Waskow; Omnia Zayed; Paul Buitelaar

REVIEW 4 major objections 5 minor 20 references

Large language models can annotate opinion spans reliably, but the relational links between spans are where they lose fidelity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:15 UTC pith:I6CAL6ZJ

load-bearing objection Useful element-wise diagnostic that LLMs get opinion spans but not relations; needs a contamination check and a cleaner IAA story before the 'high reliability' claim can stand. the 4 major comments →

arxiv 2601.16800 v4 pith:I6CAL6ZJ submitted 2026-01-23 cs.CL

Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

Gaurav Negi , MA Waskow , John McCrae , Omnia Zayed , Paul Buitelaar This is my paper

classification cs.CL

keywords LLM annotationfine-grained opinion analysisASTEACOSannotation adjudicationinter-annotator agreementsentiment analysisdata augmentation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that LLMs can serve as automatic annotators and adjudicators for fine-grained opinion analysis, cutting the cost of creating labeled datasets. Using a declarative annotation pipeline and an LLM-based adjudicator that merges multiple model annotations, the authors show the approach works well for identifying individual opinion spans—aspect terms, opinion phrases, sentiment polarity—but fails to faithfully reconstruct the relational structures, such as which opinion phrase modifies which target, especially in the more complex ACOS quadruple task. The upshot the authors draw is that LLMs are better positioned as high-fidelity annotation assistants and data augmentation tools, not replacements for human annotators.

Core claim

The central discovery is a performance bifurcation: at the span level, LLM annotators align closely with human annotations (sentiment polarity especially, with aspect and opinion spans not far behind), while at the relation level—pairing an aspect term with the opinion span that expresses it, and assigning the correct aspect category—their agreement with human labels drops sharply. Larger models (32B) align better than smaller ones, and an LLM adjudicator that combines redundant annotations from several models improves exact-match F1 in most settings, sometimes making a 4B ensemble outperform individual 14B models. On the ACOS quadruple task, the aspect category component is the main bottlen

What carries the argument

The mechanism is a declarative annotation pipeline that programs LLMs rather than hand-crafting prompts: a data model specifies input/output structure, a small set of annotated examples is used to optimise the prompt, and three LLMs of different sizes each produce a redundant annotation set. A fourth step uses one LLM as an adjudicator, taking the redundant annotations plus the text and producing final labels—an ensemble in the spirit of stacked generalisation. The pipeline is evaluated with exact-match F1 against human annotations for ASTE triplets and ACOS quadruples.

Load-bearing premise

The load-bearing premise is that high agreement among the LLM annotators signals annotation quality; without a human-human agreement baseline on the same datasets, the three models could be consistently wrong together and still show high inter-annotator agreement.

What would settle it

Compare the inter-annotator agreement of the three LLM annotators to human-human agreement on a sample of the same ASTE and ACOS test texts. If human-human agreement is substantially higher than LLM-LLM agreement on the relational components (aspect–opinion link and aspect category), the claim that LLMs are reliable annotators at the relational level is falsified. A simpler check: list cases where all three LLMs agree but all disagree with the human annotation and count their frequency.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

LLM-generated span labels can be used to bootstrap or expand opinion datasets cheaply, with human effort redirected to correcting relational links.
Adjudication of multiple LLM annotations can yield final labels that outperform the best individual annotator, especially for smaller models.
The ACOS bottleneck on aspect categories suggests that category coverage and implicit aspect handling are the priority for improving automatic annotation.
For ASTE, the pipeline is closer to production-ready, while ACOS needs more work before LLM annotations can be trusted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's reliability claim rests on inter-annotator agreement among LLMs, but if the three models share training-data biases, high agreement can coexist with systematic error; a human-human agreement baseline on the same datasets would anchor the claim.
A natural extension is to test the adjudication method on other structured annotation tasks, such as event or relation extraction, where span extraction is usually easier than link prediction.
Because the category prediction in ACOS is the main failure mode, augmenting training data with rare categories or using ontology-aware prompting could yield outsized gains.
The temperature-0 setting may understate the variability that matters in practice; a test of adjudication under sampling diversity could reveal whether the ensemble is robust to more varied annotator outputs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful element-wise diagnostic that LLMs get opinion spans but not relations; needs a contamination check and a cleaner IAA story before the 'high reliability' claim can stand.

read the letter

Colleague,

This is a useful empirical paper, not a breakthrough, and it has reporting problems that need fixing before I'd trust the conclusions. The core contribution is the element-wise breakdown: for ASTE/ACOS, LLMs get sentiment and individual spans mostly right, but the joint relations—aspect-opinion pairs, and especially full quadruples—fall apart. That span-versus-relational bifurcation is consistent across models and datasets, and it is the kind of concrete diagnostic that makes the paper worth reading. The DSPy pipeline is competently executed, and the adjudicator, while simple, does improve results most of the time.

Where I part ways with the abstract: "high Inter-Annotator Agreement across individual LLM-based annotators" is measured between LLMs, not against humans. If three models share the same biases—likely, given overlapping training data—high alpha can coexist with all three being wrong. The F1 against human gold anchors quality, so it's not fatal, but without human-human alpha on the same data, "reliable" is unanchored.

More concerning is that all evaluation uses the public ASTE/ACOS test splits from 2014–2016, while the models are from 2024–2025. The paper never checks for contamination or reports knowledge cutoffs. At temperature zero, with ICL examples drawn from the same public dev splits, the span-level F1 could partly be memory of benchmark instances. That doesn't sink the paper—the span-versus-relation pattern would still need explaining—but it does mean the "high-fidelity annotation assistant for new text" claim is not actually measured.

The abstract also promises a benchmark against exact, flexible, and element-wise rule-based voting aggregators, and I could not find that comparison anywhere in the body. That is a missing promised result. There are also small but real errors: Table 6 has an ACOS value of 36.75 that should be 0.3675, and one F1 in Table 2 doesn't match its P and R. No error bars or significance tests either, though with temperature zero that is less critical.

Net: the main diagnostic is solid enough to build on, but the paper overstates reliability and under-delivers on its own promised comparison. I would send it to peer review, but only with a request that the authors run a contamination check, add a human-human IAA baseline, fix the missing rule-based table, and tone down the reliability language.

Best, [You]

Referee Report

4 major / 5 minor

Summary. The paper investigates the use of LLMs as automatic annotators and adjudicators for fine-grained opinion analysis, specifically for Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) quadruple extraction. It proposes a declarative DSPy-based annotation pipeline with multiple LLMs as annotators and a separate LLM as an adjudicator that merges redundant annotations. Experiments on six public benchmark datasets across three model-size classes report exact-match and element-wise F1 scores against human gold labels, plus Krippendorff α among the LLM annotators. The central claim is that LLMs achieve high inter-annotator agreement and are reliable at identifying opinion spans, but struggle to reproduce the relational structures linking spans, positioning LLMs as annotation assistants rather than full replacements for human annotators.

Significance. If valid, the result would support a practical pipeline for lowering the cost of creating fine-grained opinion datasets, with humans verifying relational links rather than annotating from scratch. The paper has notable strengths: it evaluates across multiple model families and sizes, provides element-wise F1 decompositions that localize errors to specific span relations, reports IAA for LLM annotators, and includes a qualitative error analysis. However, the central generalization claim is currently under-supported because the evaluation uses public benchmark test splits that predate model training, the IAA result lacks any human-human baseline, and the abstract promises a rule-based voting comparison that never appears in the body. These issues are fixable within the scope of the paper, but they are load-bearing for the stated conclusions.

major comments (4)

[Abstract; Sections 2–7] The abstract states that the adjudication methodology is 'benchmarked against exact, flexible, and element-wise variants of a rule-based voting aggregator,' but no such rule-based voting baseline appears anywhere in the body. Tables 2 and 3 compare only individual annotators and the LLM adjudicator. This is a substantive missing comparison for the adjudication contribution; either add the baseline or revise the abstract.
[§3.2, §6, Tables 2–5] The test splits come from SemEval 2014–2016 (lap14, res14, res15, res16) and the ACOS splits from the same period, while all evaluated models (Qwen3, MiniCPM3, Phi-4, DeepSeek-R1, gpt-oss) were released in 2024–2025. No model knowledge cutoffs are reported and no contamination check is performed. Because temperature is 0 and in-context examples are drawn from the same public dev splits, high span-level F1 could reflect memorization of public benchmark instances rather than an ability to annotate unseen opinion text. Since the abstract and conclusion frame the contribution as reducing annotation cost for new, domain-specific data, this is load-bearing. Please report knowledge cutoffs, test for overlap, or evaluate on text released after model training.
[§7.1, Table 6] The reliability claim is based on Krippendorff α among the three LLM annotators. If the LLMs share systematic biases—likely given overlapping training data—high LLM-LLM agreement can occur while all three disagree with humans. No human-human α on the same datasets is reported, so 'highly reliable' has no external anchor. Additionally, the computation of α for overlapping span annotations is not described (unitizing, annotation units, software used). Please report human-human agreement or validate LLM agreement against human labels.
[§5, §6] The adjudicator is the best-performing annotator, A1, and the input to adjudication includes A1's own annotations. This self-referential design can inflate adjudicated agreement relative to a scenario where the adjudicator is independent of the annotators. The effect is not quantified. Please either use an independent adjudicator, remove A1's output from the adjudicator input, or analyze the extent of this inflation.

minor comments (5)

[Table 6] The bottom half of Table 6 is mislabeled as 'Performance scores for ACOS tasks' but the values appear to be Krippendorff α values. In addition, the Mini laptop row contains '36.75' where an α value between 0 and 1 is expected; this appears to be a typographical error.
[§8] The paper spells 'Krippendorff' as 'Kirppendorff' in the conclusion. Please correct.
[§5] The annotator assignment line reads 'A1, A1, A3' instead of 'A1, A2, A3'.
[§7.2] In the error analysis discussion, the text says 'a minor mistake in identifying the aspect term (ac)' — in ASTE the aspect term is abbreviated 'at', not 'ac', which is the ACOS category abbreviation.
[General] No code, prompts, or data are released. Given the claim of a declarative DSPy pipeline, releasing the optimized prompts and scripts would materially improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity; the paper's central claims are measured against external human gold labels.

full rationale

The paper's core claims are empirical evaluations against human-annotated gold standards from the public ASTE and ACOS test splits (§3.2, §6). There is no fitted parameter that is then renamed as a prediction: the only optimization is DSPy prompt selection on development data, evaluated on held-out test data (§4.2, §5). The adjudication method is benchmarked against rule-based voting aggregators, and its output is scored against human labels; it is not defined in terms of the target result. The IAA (Krippendorff α) among LLM annotators is reported as a reliability measure, but the central span-level accuracy claim rests on element-wise F1 against human annotations (Tables 4-5), not on LLM-LLM agreement. The absence of human-human α is an external-validity limitation, not a circular step. The fact that the best-performing annotator A1 also serves as adjudicator is an experimental design choice; it does not reduce the adjudication result to A1's own outputs by construction, and it is not used as evidence for the annotation-generality claim. No load-bearing self-citations were found; all cited prior work is external. Overall, the derivation chain is self-contained with respect to its evaluation target, so the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new theoretical entities or mechanisms are introduced. The 'LLM adjudicator' is an existing model playing a role, not a new entity. The ledger's main burden is the human-ground-truth assumption and the ICL/annotator-selection choices.

free parameters (4)

Number of in-context examples = 5, 10, or 15 (selected per model-task on dev subset)
The best ICL count is chosen by performance on the evaluation subset (Section 5); it is not derived from theory.
Annotator role assignment = A1/A2/A3 model assignment per size group
Models are assigned to annotator roles based on evaluation-sample performance (Section 5), which affects which outputs are adjudicated.
Adjudicator choice = A1 (best-performing annotator)
The adjudicator is chosen as the best annotator using dev performance (Section 5), making adjudication gains dependent on this selection.
DSPy-optimized prompt text = Not released
DSPy compiles prompts from ICL candidates (Section 4.2); the final prompt content is optimized on data and not provided, limiting reproducibility.

axioms (5)

domain assumption Human gold labels from ASTE/ACOS datasets are treated as error-free ground truth
F1 alignment with human labels is the sole quality measure (Section 6).
domain assumption Krippendorff α among three LLM annotators measures annotation reliability
Section 7.1; high LLM-LLM agreement may reflect shared biases rather than correctness.
domain assumption Exact-match F1 (and element-wise spans) captures annotation quality
Sections 5–6; no partial-credit or distance-based metrics are used.
domain assumption DSPy prompt optimization on a dev subset transfers to the test set without overfitting
Sections 4.2 and 5; no multi-seed or cross-validation checks are reported.
domain assumption ASTE/ACOS category taxonomies are closed-set and require faithful reproduction
Section 3.2; models must predict categories like the 114 laptop-domain ACOS categories from text without additional supervision.

pith-pipeline@v1.3.0-alltime-deepseek · 11447 in / 15173 out tokens · 159194 ms · 2026-08-04T06:15:32.906980+00:00 · methodology

0 comments

read the original abstract

Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments and their targets. Although this level of detail is valuable, annotating opinions in datasets for model training requires considerable human effort and substantial cost, especially across diverse domains and real-world applications. To address this shortage of domain-specific labelled datasets, we explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis. We use a declarative annotation pipeline, an approach that reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a dedicated methodology for an LLM to adjudicate multiple labels and produce final annotations, benchmarked against exact, flexible, and element-wise variants of a rule-based voting aggregator. We trial the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks. Our results reveal a critical performance bifurcation: LLMs are reliable at the span level yet struggle to reproduce the relational structures that connect those spans faithfully. This suggests that LLMs are better positioned as high-fidelity annotation assistants and data augmentation tools to expand fine-grained opinion-annotated datasets, rather than replacing human annotators entirely.

Figures

Figures reproduced from arXiv: 2601.16800 by Gaurav Negi, John McCrae, MA Waskow, Omnia Zayed, Paul Buitelaar.

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith

[1]

In: Zong, C., Xia, F., Li, W., Navigli, R

Cai, H., Xia, R., Yu, J.: Aspect-category-opinion-sentiment quadruple extraction with implicit aspects and opinions. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers)...

doi:10.18653/v1/2021.acl- 2021
[2]

In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=b0y6fbSUG0

Hao, S., Gu, Y., Luo, H., Liu, T., Shao, X., Wang, X., Xie, S., Ma, H., Samavedhi, A., Gao, Q., Wang, Z., Hu, Z.: LLM reasoners: New evaluation, library, and anal- ysis of step-by-step reasoning with large language models. In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=b0y6fbSUG0

2024
[3]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Hasanain, M., Ahmad, F., Alam, F.: Large language models for pro- paganda span annotation. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP
[4]

In: Kim, W., Kohavi, R., Gehrke, J., DuMouchel, W

Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Kim, W., Kohavi, R., Gehrke, J., DuMouchel, W. (eds.) Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, Seattle, Washington, USA, August 22-25, 2004. pp. 168–177. ACM (2004). https://doi.org/10.1145/1014052.1014073, https://doi.org/10.1145/10...

arXiv 2004
[5]

arXiv preprint arXiv:2504.08697 (2025)

Kasner, Z., Zouhar, V., Schmidtová, P., Kartáč, I., Onderková, K., Plátek, O., Gkatzia, D., Mahamood, S., Dušek, O., Balloccu, S.: Large language models as span annotators. arXiv preprint arXiv:2504.08697 (2025)

arXiv 2025
[6]

In: ICLR (2024)

Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vard- hamanan, S., Haq, S., Sharma, A., Joshi, T.T., Moazam, H., Miller, H., Zaharia, M., Potts, C.: Dspy: Compiling declarative language model calls into self-improving pipelines. In: ICLR (2024)

2024
[7]

Computational Linguistics50(3), 817–866 (Sep 2024)

Klie, J.C., Eckart de Castilho, R., Gurevych, I.: Analyzing dataset annotation qual- ity management in the wild. Computational Linguistics50(3), 817–866 (Sep 2024). https://doi.org/10.1162/coli_a_00516, https://aclanthology.org/2024.cl-3.1/

doi:10.1162/coli_a_00516 2024
[8]

In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large lan- guage models are zero-shot reasoners. In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Informa- tion Processing Systems. vol. 35, pp. 22199–22213. Curran Associates, Inc. 12 G. Negi et al. (2022), https://proceedings.neurips.cc/paper_file...

2022
[9]

In: Proceedings of the 29th Symposium on Operating Systems Principles.p.611–626.SOSP’23,AssociationforComputingMachinery,NewYork, NY, USA (2023)

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the 29th Symposium on Operating Systems Principles.p.611–626.SOSP’23,AssociationforComputingMachinery,NewYork, NY, USA (2023). https://doi.org/10.1145/360000...

arXiv 2023
[10]

In: Ag- garwal, C.C., Zhai, C

Liu, B., Zhang, L.: A Survey of Opinion Mining and Sentiment Analysis. In: Ag- garwal, C.C., Zhai, C. (eds.) Mining Text Data, pp. 415–463. Springer US, Boston, MA (2012). https://doi.org/10.1007/978-1-4614-3223-4_13

doi:10.1007/978-1-4614-3223-4_13 2012
[11]

(eds.) Robust Argumentation Machines - First International Con- ference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024, Proceedings

Mirzakhmedova, N., Gohsen, M., Chang, C., Stein, B.: Are large language mod- els reliable argument quality annotators? In: Cimiano, P., Frank, A., Kohlhase, M., Stein, B. (eds.) Robust Argumentation Machines - First International Con- ference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024, Proceedings. Lec- ture Notes in Computer Science, vol. 14638, pp....

doi:10.1007/978-3-031-63536-6 2024
[12]

Proceedings of the AAAI Conference on Artificial Intelli- gence34(05), 8600–8607 (Apr 2020)

Peng, H., Xu, L., Bing, L., Huang, F., Lu, W., Si, L.: Knowing what, how and why: A near complete solution for aspect-based senti- ment analysis. Proceedings of the AAAI Conference on Artificial Intelli- gence34(05), 8600–8607 (Apr 2020). https://doi.org/10.1609/aaai.v34i05.6383, https://ojs.aaai.org/index.php/AAAI/article/view/6383

doi:10.1609/aaai.v34i05.6383 2020
[13]

In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

Toprak, C., Jakob, N., Gurevych, I.: Sentence and expression level annotation of opinions in user-generated discourse. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. p. 575–584. ACL ’10, Association for Computational Linguistics, USA (2010)

2010
[14]

CoRRabs/2503.16974(2025)

Wang, J.J., Wang, V.X.: Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. CoRRabs/2503.16974(2025). https://doi.org/10.48550/ARXIV.2503.16974

Pith/arXiv arXiv doi:10.48550/arxiv.2503.16974 2025
[15]

Neural Netw.5(2), 241–259 (Feb 1992)

Wolpert, D.H.: Original contribution: Stacked generalization. Neural Netw.5(2), 241–259 (Feb 1992). https://doi.org/10.1016/S0893-6080(05)80023-1

doi:10.1016/s0893-6080(05)80023-1 1992
[16]

CoRRabs/2010.04640(2020), https://arxiv.org/abs/2010.04640

Wu, Z., Ying, C., Zhao, F., Fan, Z., Dai, X., Xia, R.: Grid tagging scheme for aspect-oriented fine-grained opinion extraction. CoRRabs/2010.04640(2020), https://arxiv.org/abs/2010.04640

Pith/arXiv arXiv 2010
[17]

In: Webber, B., Cohn, T., He, Y., Liu, Y

Xu, L., Li, H., Lu, W., Bing, L.: Position-aware tagging for aspect sentiment triplet extraction. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,Online,November16-20,2020.pp.2339–2349.AssociationforComputational Linguistics (2020). https://doi.org/10.18653...

doi:10.18653/v1/2020.emnlp-main.183 2020
[18]

IEEE Transac- tions on Knowledge and Data Engineering35(11), 11019–11038 (2023)

Zhang, W., Li, X., Deng, Y., Bing, L., Lam, W.: A survey on aspect- based sentiment analysis: Tasks, methods, and challenges. IEEE Transac- tions on Knowledge and Data Engineering35(11), 11019–11038 (2023). https://doi.org/10.1109/TKDE.2022.3230975

arXiv 2023
[19]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., Chen, K.: ProSA: Assessing and understanding the prompt sensitivity of LLMs. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 1950–1976. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10....

doi:10.18653/v1/2024.findings-emnlp.108 2024
[2024]

14522–14532

pp. 14522–14532. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10.18653/v1/2024.findings-emnlp.850, https://aclanthology.org/2024.findings-emnlp.850/

doi:10.18653/v1/2024.findings-emnlp.850 2024

[1] [1]

In: Zong, C., Xia, F., Li, W., Navigli, R

Cai, H., Xia, R., Yu, J.: Aspect-category-opinion-sentiment quadruple extraction with implicit aspects and opinions. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers)...

doi:10.18653/v1/2021.acl- 2021

[2] [2]

In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=b0y6fbSUG0

Hao, S., Gu, Y., Luo, H., Liu, T., Shao, X., Wang, X., Xie, S., Ma, H., Samavedhi, A., Gao, Q., Wang, Z., Hu, Z.: LLM reasoners: New evaluation, library, and anal- ysis of step-by-step reasoning with large language models. In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=b0y6fbSUG0

2024

[3] [3]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Hasanain, M., Ahmad, F., Alam, F.: Large language models for pro- paganda span annotation. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP

[4] [4]

In: Kim, W., Kohavi, R., Gehrke, J., DuMouchel, W

Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Kim, W., Kohavi, R., Gehrke, J., DuMouchel, W. (eds.) Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, Seattle, Washington, USA, August 22-25, 2004. pp. 168–177. ACM (2004). https://doi.org/10.1145/1014052.1014073, https://doi.org/10.1145/10...

arXiv 2004

[5] [5]

arXiv preprint arXiv:2504.08697 (2025)

Kasner, Z., Zouhar, V., Schmidtová, P., Kartáč, I., Onderková, K., Plátek, O., Gkatzia, D., Mahamood, S., Dušek, O., Balloccu, S.: Large language models as span annotators. arXiv preprint arXiv:2504.08697 (2025)

arXiv 2025

[6] [6]

In: ICLR (2024)

Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vard- hamanan, S., Haq, S., Sharma, A., Joshi, T.T., Moazam, H., Miller, H., Zaharia, M., Potts, C.: Dspy: Compiling declarative language model calls into self-improving pipelines. In: ICLR (2024)

2024

[7] [7]

Computational Linguistics50(3), 817–866 (Sep 2024)

Klie, J.C., Eckart de Castilho, R., Gurevych, I.: Analyzing dataset annotation qual- ity management in the wild. Computational Linguistics50(3), 817–866 (Sep 2024). https://doi.org/10.1162/coli_a_00516, https://aclanthology.org/2024.cl-3.1/

doi:10.1162/coli_a_00516 2024

[8] [8]

In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large lan- guage models are zero-shot reasoners. In: Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Informa- tion Processing Systems. vol. 35, pp. 22199–22213. Curran Associates, Inc. 12 G. Negi et al. (2022), https://proceedings.neurips.cc/paper_file...

2022

[9] [9]

In: Proceedings of the 29th Symposium on Operating Systems Principles.p.611–626.SOSP’23,AssociationforComputingMachinery,NewYork, NY, USA (2023)

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the 29th Symposium on Operating Systems Principles.p.611–626.SOSP’23,AssociationforComputingMachinery,NewYork, NY, USA (2023). https://doi.org/10.1145/360000...

arXiv 2023

[10] [10]

In: Ag- garwal, C.C., Zhai, C

Liu, B., Zhang, L.: A Survey of Opinion Mining and Sentiment Analysis. In: Ag- garwal, C.C., Zhai, C. (eds.) Mining Text Data, pp. 415–463. Springer US, Boston, MA (2012). https://doi.org/10.1007/978-1-4614-3223-4_13

doi:10.1007/978-1-4614-3223-4_13 2012

[11] [11]

(eds.) Robust Argumentation Machines - First International Con- ference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024, Proceedings

Mirzakhmedova, N., Gohsen, M., Chang, C., Stein, B.: Are large language mod- els reliable argument quality annotators? In: Cimiano, P., Frank, A., Kohlhase, M., Stein, B. (eds.) Robust Argumentation Machines - First International Con- ference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024, Proceedings. Lec- ture Notes in Computer Science, vol. 14638, pp....

doi:10.1007/978-3-031-63536-6 2024

[12] [12]

Proceedings of the AAAI Conference on Artificial Intelli- gence34(05), 8600–8607 (Apr 2020)

Peng, H., Xu, L., Bing, L., Huang, F., Lu, W., Si, L.: Knowing what, how and why: A near complete solution for aspect-based senti- ment analysis. Proceedings of the AAAI Conference on Artificial Intelli- gence34(05), 8600–8607 (Apr 2020). https://doi.org/10.1609/aaai.v34i05.6383, https://ojs.aaai.org/index.php/AAAI/article/view/6383

doi:10.1609/aaai.v34i05.6383 2020

[13] [13]

In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

Toprak, C., Jakob, N., Gurevych, I.: Sentence and expression level annotation of opinions in user-generated discourse. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. p. 575–584. ACL ’10, Association for Computational Linguistics, USA (2010)

2010

[14] [14]

CoRRabs/2503.16974(2025)

Wang, J.J., Wang, V.X.: Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. CoRRabs/2503.16974(2025). https://doi.org/10.48550/ARXIV.2503.16974

Pith/arXiv arXiv doi:10.48550/arxiv.2503.16974 2025

[15] [15]

Neural Netw.5(2), 241–259 (Feb 1992)

Wolpert, D.H.: Original contribution: Stacked generalization. Neural Netw.5(2), 241–259 (Feb 1992). https://doi.org/10.1016/S0893-6080(05)80023-1

doi:10.1016/s0893-6080(05)80023-1 1992

[16] [16]

CoRRabs/2010.04640(2020), https://arxiv.org/abs/2010.04640

Wu, Z., Ying, C., Zhao, F., Fan, Z., Dai, X., Xia, R.: Grid tagging scheme for aspect-oriented fine-grained opinion extraction. CoRRabs/2010.04640(2020), https://arxiv.org/abs/2010.04640

Pith/arXiv arXiv 2010

[17] [17]

In: Webber, B., Cohn, T., He, Y., Liu, Y

Xu, L., Li, H., Lu, W., Bing, L.: Position-aware tagging for aspect sentiment triplet extraction. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,Online,November16-20,2020.pp.2339–2349.AssociationforComputational Linguistics (2020). https://doi.org/10.18653...

doi:10.18653/v1/2020.emnlp-main.183 2020

[18] [18]

IEEE Transac- tions on Knowledge and Data Engineering35(11), 11019–11038 (2023)

Zhang, W., Li, X., Deng, Y., Bing, L., Lam, W.: A survey on aspect- based sentiment analysis: Tasks, methods, and challenges. IEEE Transac- tions on Knowledge and Data Engineering35(11), 11019–11038 (2023). https://doi.org/10.1109/TKDE.2022.3230975

arXiv 2023

[19] [19]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., Chen, K.: ProSA: Assessing and understanding the prompt sensitivity of LLMs. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 1950–1976. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10....

doi:10.18653/v1/2024.findings-emnlp.108 2024

[20] [2024]

14522–14532

pp. 14522–14532. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10.18653/v1/2024.findings-emnlp.850, https://aclanthology.org/2024.findings-emnlp.850/

doi:10.18653/v1/2024.findings-emnlp.850 2024