{"id":"896c7974-c703-43c4-90ff-1ca1ea65fed5","arxiv_id":"2508.08125","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 3.1K-review Czech ABSA dataset with linked target-category-polarity annotations in SemEval-2016 format, plus 24M unlabeled reviews and Transformer baselines.","lead":"Researchers created a new Czech-language dataset of 3,100 restaurant reviews annotated for aspect-based sentiment analysis. It is built to support complex opinion-mining tasks and to allow direct comparison with English SemEval-2016 data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset value hinges on linked-annotation quality, but '~90% agreement' is raw percentage; without chance-corrected kappa or per-category agreement, the reliability of the target–aspect-category links is unestablished.","rationale":"The reader's weakest assumption is that the reported ~90% inter-annotator agreement reflects genuine label correctness rather than chance agreement or shared misinterpretation. My concern is the same: raw percentage agreement is a weak and potentially misleading reliability metric for multi-label, imbalanced ABSA annotations. The paper's central claim—that this is a usable complex-task benchmark—depends on the linked annotations being trustworthy. Without kappa or per-category agreement, the 90% figure cannot carry that weight. This is not an internal inconsistency or a challenge to consensus; it is a missing piece of evidence that the abstract itself does not supply. The concrete test is feasible: if per-annotator judgments exist, kappa can be computed from the released dataset; if not, an independent re-annotation sample would settle it. This does not move the reader's CONDITIONAL verdict; it sharpens the condition the authors must meet. I therefore recommend UNCHANGED.","tokens_in":9122,"tokens_out":5166,"duration_ms":64312,"concrete_test":"Compute Cohen's kappa (or Krippendorff's alpha) from the two annotators' raw judgments for each annotation layer, especially the target–aspect-category link tuples, and report per-category agreement. If per-annotator labels are not released, re-annotate a stratified random sample of 100–200 reviews with an independent annotator using the published guidelines and compute agreement on exact linked tuples. If kappa on the link layer is below 0.7, the '~90% agreement' claim is insufficient and the dataset's complex-task reliability is not demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is a benchmark whose utility depends on correct, consistent, and format-conformant linked annotations. The only reliability evidence reported is 'approximately 90%' inter-annotator agreement. In ABSA annotation, raw percentage agreement is inflated by label imbalance: if dominant categories such as 'no aspect' or 'FOOD#POSITIVE' occur frequently, two annotators can agree by chance on most units. The abstract provides no kappa, no per-layer agreement (target spans, category labels, polarity, link existence), no adjudication protocol, and no annotation-guideline excerpts. Thus the 90% figure does not establish that the linked labels are reliable enough to support the reported baselines or cross-lingual comparisons. A related risk is that the unified linked format may have been constructed by converting separate labels from the predecessor dataset rather than by independent linked annotation; if so, alignment errors would be invisible in raw agreement. Both issues point to the same need: chance-corrected, per-layer, per-category reliability evidence for the linked annotation layer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new Czech ABSA dataset consisting of 3.1K manually annotated restaurant reviews, built on an older Czech ABSA corpus and reformatted into the SemEval-2016 unified annotation format to support complex linked tasks such as target-aspect-category detection. The authors report approximately 90% raw inter-annotator agreement, release 24M unlabeled reviews, and provide Transformer-based monolingual baselines with error analysis. The dataset and code are made available for non-commercial research.","tokens_in":9291,"tokens_out":2790,"duration_ms":33283,"significance":"If the annotation quality and format claims hold, this is a useful resource: it provides the first Czech ABSA dataset in a SemEval-2016-compatible format, enabling cross-lingual comparison and evaluation of linked ABSA tasks in Czech. The release of 24M unlabeled reviews is a substantial additional resource for unsupervised and semi-supervised methods. The inclusion of multiple Transformer baselines is a practical benefit, and the public release supports reproducibility.","major_comments":[{"comment":"The only reliability evidence reported for the linked annotations is 'an inter-annotator agreement rate of approximately 90%'. This is a raw percentage, which is known to be inflated by label imbalance in ABSA annotation. The paper must report chance-corrected agreement (e.g., Cohen's kappa or Krippendorff's alpha) and per-layer/per-category agreement for target spans, aspect categories, polarities, and especially the target-aspect-category link. Without this, the reported baseline scores and the claimed cross-lingual comparability rest on an unverified quality assertion.","section":"Abstract"},{"comment":"The dataset is built on an older Czech dataset that contained separate labels for basic ABSA tasks, and the new contribution is the unified linked format. The manuscript must clarify whether the links were produced by fresh annotation of the unified format or by mechanically combining the predecessor's separate labels. If derived mechanically, link-alignment errors would be invisible to the reported agreement. Please provide annotation guidelines, adjudication protocol, and a worked example showing how target, aspect category, and polarity are linked.","section":"Abstract / annotation procedure"},{"comment":"The submitted full text is heavily corrupted by an encoding issue, leaving most of the paper unreadable. I cannot verify the dataset statistics, annotation details, baseline configurations, hyperparameters, or the error analysis. This is not a cosmetic defect; it makes the technical content uncheckable. A clean, properly encoded manuscript is required for any further review.","section":"Full text (all sections)"}],"minor_comments":[{"comment":"The abstract says '24M reviews without annotations'; please clarify whether this counts raw review documents or individual sentences/reviews and whether these unlabeled reviews come from the same domain and time period as the 3.1K annotated set.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the core deliverable is a dataset, so the review hinges on annotation reliability and format fidelity. The abstract's raw 90% agreement is not sufficient evidence for linked annotations. The unreadable full text further prevents verification. I see no derivation-versus-input circularity, but the manuscript needs a clean resubmission and stronger reliability statistics before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real product here is a linked-annotation Czech ABSA benchmark in SemEval-2016 format. That is genuinely new: the predecessor dataset had separate labels for basic tasks, and this one unifies them for target-aspect-category detection and other complex tasks. Format compatibility is a concrete plus, since it enables direct comparison with English and other SemEval-2016 resources. The paper also delivers 3.1K manually annotated reviews, 24M unlabeled reviews, and Transformer baselines with error analysis. As a resource contribution, this is solid.\n\nThe main soft spot, which the stress-test note gets right, is reliability evidence for the linked layer. Raw 90% agreement is inflated by label imbalance in ABSA, and the linked target-aspect-category annotations are exactly where alignment errors would hide. There is no kappa, no per-category or per-layer agreement, no adjudication protocol, and no guideline excerpts in the abstract. If the authors converted older labels rather than independently annotating the links, raw agreement would not catch conversion mistakes. This is a fixable problem: report chance-corrected agreement, break it down by layer and category, and describe the linking process.\n\nOne caveat: the full text in the copy I had was unreadable mojibake, so I could not verify baseline numbers, split construction, license details, or the download link. The reader's conditional verdict is appropriate. Nothing in the abstract suggests a defect, but the verification burden is on the authors.\n\nThis paper is for the ABSA and lower-resource NLP communities. It deserves a serious referee, not a desk reject. My recommendation: send it to peer review with the explicit request that the authors supply kappa, per-layer agreement, and a transparent description of how the linked annotations were produced and adjudicated. The resource is worth having, but its usability depends on that evidence.","headline":"Genuinely useful Czech ABSA resource; the linked-annotation layer is the new contribution, but the 90% agreement figure needs kappa and per-layer evidence before cross-lingual claims lean on it.","tokens_in":591,"tokens_out":1145,"would_cite":true,"duration_ms":31391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 3.1K-review Czech benchmark puts aspect-based sentiment analysis into the SemEval-2016 linked format, enabling cross-lingual comparisons.","keywords":["aspect-based sentiment analysis","Czech dataset","SemEval-2016","target-aspect-category detection","restaurant reviews","cross-lingual evaluation","annotation corpus","unsupervised learning"],"falsifier":"Take a random sample of a few hundred released reviews, have a fresh pair of annotators label them from scratch using the published annotation scheme, and compare against the released annotations; if pairwise agreement on the full linked target-aspect-category tuples is much below 90% or chance-corrected agreement is low, the claimed reliability and conversion fidelity are not supported.","tokens_in":8987,"feed_emoji":"🍽️","tokens_out":5038,"duration_ms":59718,"temperature":0.7,"pith_summary":"This paper aims to establish that a Czech dataset can support the complex, linked version of aspect-based sentiment analysis (ABSA), not just the basic subtasks. To do that, it converts an older Czech restaurant-review corpus into the SemEval-2016 format, where each aspect term, category, and polarity is explicitly connected, and adds 3.1K fresh annotations with about 90% inter-annotator agreement. If the dataset is sound, it gives Czech a benchmark that can be plugged into cross-lingual comparisons and used to train models on tasks such as target-aspect-category detection. The 24M unlabeled Czech reviews are a secondary resource for unsupervised learning.","feed_headline":"Czech dataset links aspects, targets, and sentiments in 3.1K reviews","feed_subtitle":"Hand-annotated restaurant reviews in SemEval-2016 format set up cross-language ABSA comparisons.","key_machinery":"The load-bearing object is the SemEval-2016-style linked annotation record. Instead of storing aspect terms, categories, and polarities as independent labels, each review is annotated with a single unified structure that explicitly connects a target mention to its aspect category and its sentiment polarity. That linkage is what turns the dataset from a collection of basic ABSA subtasks into a resource for complex tasks such as target-aspect-category detection, and it is also what makes the corpus directly comparable, format-wise, with equivalent datasets in other languages.","core_discovery":"The paper's central claim is that a Czech-language ABSA benchmark can support complex linked sentiment tasks, not just the separate basic subtasks. It presents 3.1K manually annotated restaurant reviews in which aspect terms, aspect categories, and polarity are annotated in a unified SemEval-2016-style format, so that each sentiment element is linked to the others. The corpus is derived from an older Czech dataset that only had separate labels for basic tasks, and the new format is what allows tasks such as target-aspect-category detection. Two trained annotators reached about 90% agreement, and the paper reports Transformer-based monolingual baselines plus an error analysis. It also makes 2","pith_inferences":["Beyond the paper: the linked format opens a direct route to cross-lingual transfer tests between Czech and the other SemEval-2016 languages, which this paper itself does not report.","Beyond the paper: the 24M unlabeled reviews could support domain-adaptive pretraining; a testable extension is to measure whether pretraining on them improves linked ABSA scores beyond the reported baselines.","Beyond the paper: re-annotating a subset of the original reviews with the new scheme and checking label-by-label agreement would quantify how much of the new resource's value comes from the linked format versus fresh annotation work."],"forward_implications":["Same-format Czech data means systems trained or evaluated on SemEval-2016 English data can be run on Czech without reformatting.","Linked target-aspect-category annotations enable joint models rather than separate extraction-then-polarity pipelines.","Published Transformer baselines give reference numbers for future Czech ABSA work.","The 24M unlabeled Czech reviews can support unsupervised pretraining or domain adaptation."],"supporting_citations":[],"fun_headline_variants":["Czech ABSA dataset: 3.1K reviews with linked aspects, targets, sentiments","3.1K Czech reviews unite aspect terms, categories, and polarity","New Czech corpus for complex sentiment tasks: linked labels in 3.1K reviews","SemEval-2016-format Czech dataset links aspects, targets, sentiments"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The annotations are only as good as the human labelling: that the roughly 90% agreement reflects true correctness, and that converting the older dataset's separate labels into one linked SemEval-style format did not change what was originally annotated.","fun_headline_variants_meta":{"raw":{"variants":["Czech ABSA dataset: 3.1K reviews with linked aspects, targets, sentiments","3.1K Czech reviews unite aspect terms, categories, and polarity","New Czech corpus for complex sentiment tasks: linked labels in 3.1K reviews","SemEval-2016-format Czech dataset links aspects, targets, sentiments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000913,"raw_usage":{"total_tokens":3752,"prompt_tokens":729,"completion_tokens":3023,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":2935}},"tokens_in":473,"tokens_out":3023,"duration_ms":25164,"temperature":1.0,"reasoning_tokens":2935,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:37:28.949389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of a few hundred released reviews, have a fresh pair of annotators label them from scratch using the published annotation scheme, and compare against the released annotations; if pairwise agreement on the full linked target-aspect-category tuples is much below 90% or chance-corrected agreement is low, the claimed reliability and conversion fidelity are not supported.","supporting_citations":[],"review_version":1}