{"id":"bc9d1b2f-2777-43b8-8ce8-6db236b059ca","arxiv_id":"2412.05964","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic map and benchmark of Turkish sentiment analysis datasets shows that off-the-shelf model performance depends heavily on dataset characteristics.","lead":"This paper reviews ten years of Turkish sentiment analysis research, compiling 23 datasets and labeling 31 studies, then runs four existing sentiment models on four popular datasets. It finds that model accuracy varies a lot with the dataset, so a model's score on one kind of Turkish text does not predict its score on another.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's metric triples are internally inconsistent under the paper's own evaluation protocol, so the numerical support for the headline claim is not yet verifiable.","rationale":"The reader's weakest assumption was that the 'Both: Binary' protocol, which excludes neutral model predictions, could selectively inflate accuracy and is not fully specified. My reading agrees that the protocol is problematic, but I locate the issue more precisely: Table 4 contains combinations of accuracy and recall values that are impossible under standard definitions for balanced binary datasets, regardless of whether neutral predictions are excluded. This is a stronger concern than mere under-specification because it suggests the metric computations themselves are not reliable. The central claim, that model performance depends on target text characteristics, rests primarily on Table 4. If the numbers cannot be reproduced, the claimed cross-dataset differences could be artifacts of the evaluation procedure rather than properties of the datasets. At the same time, the raw accuracy columns alone show large differences, so the qualitative finding may survive a corrected analysis. The appropriate response is therefore to keep the CONDITIONAL verdict and require the authors to release exact per-instance predictions and metric code, then verify all Table 4 entries. I do not see grounds to reject outright, because the issue is plausibly fixable by recomputation and the paper's taxonomic/resource contributions are independent of the exact metric values.","tokens_in":15355,"tokens_out":8720,"duration_ms":84445,"concrete_test":"Clone https://github.com/sevvalckc/Turkish-SAD and run the released prediction and metric scripts on the Twt and Humir splits, capturing per-instance predictions. Recompute all Table 4 columns under three explicit protocols: (1) all predictions kept, neutral treated as incorrect; (2) neutral predictions dropped and metrics recomputed on the remaining instances; (3) neutral predictions dropped and any gold neutral instances also dropped. Check whether any protocol reproduces the XLM-T/Twt 'Both: Binary' row (Accuracy .92, Micro Recall .95) and the raw XLM-T/Twt row (Accuracy .83, Macro Recall .55) simultaneously. If none does, the reported metrics are not reproducible and the central comparison must be re-run before the claim can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper states in Section 2.1 that Twt and Humir are balanced binary datasets (1,000/1,000, and 26,700/26,700 plus 5,800/5,800). Under standard definitions, macro-recall equals accuracy on a balanced binary gold set because both equal (TP+TN)/(2P). Yet the raw 'Model: Multi, Dataset: Binary' rows report Accuracy .83 vs Macro Recall .55 for XLM-T/Twt, .59 vs .44 for TSAM/Twt, and .97 vs .65 for TSAM/Humir. These cannot all be correct. The 'Both: Binary' rows are also unreproducible under the stated protocol: if neutral predictions are removed before scoring, accuracy, micro-recall, micro-precision, and micro-F1 on the remaining instances are identical, but Table 4 lists values such as Accuracy .92 vs Micro Recall .95 for XLM-T/Twt. If instead neutral predictions are only removed from the prediction set, micro-recall should be no greater than accuracy, which is also contradicted. The paper does not define how neutral outputs enter recall and accuracy, and it does not provide per-instance predictions in the text. Since the central claim is supported by these numbers, the empirical basis is not verifiable as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compiles 31 Turkish sentiment analysis studies published between 2012 and 2022 and 23 associated datasets, labels the studies with the Rodrigues et al. taxonomy, and presents the resulting map in Tables 1–3. The authors then run four pre-trained sentiment models (XLM-T, BERTurk with BounTi, TSAM, and TurkishBERTweet) on four recurrent datasets (Twt, Turkmenoglu, TMRD-TMPR, Humir) and report accuracy, recall, precision, and F1 scores under three classification-format settings in Table 4. The central claim, stated in the abstract, is that the performance of sentiment analysis tools significantly depends on the characteristics of the target text. The paper also provides a GitHub repository with code and predictions.","tokens_in":15591,"tokens_out":6957,"duration_ms":67653,"significance":"If the reported benchmark numbers are correct and the evaluation protocol is fully specified, this study would be a useful resource for Turkish sentiment analysis: it collects and organizes a substantial set of datasets, provides a taxonomic overview of a decade of work, and offers a cross-dataset comparison using modern pre-trained models. The public availability of the code and data is a clear strength, as is the effort to compile datasets that were previously scattered or only available by request. However, as written, the central empirical claim rests on Table 4, whose evaluation protocol is not defined precisely enough to allow reproduction or verification. The paper does not currently support the strong claim of statistical or causal dependence on dataset characteristics, because the datasets differ on many axes simultaneously and no controlled comparison or significance test is provided.","major_comments":[{"comment":"The evaluation protocol for Table 4 is not defined precisely enough to be reproducible. For the 'Model: Multi, Dataset: Binary' rows, the paper states that scores are calculated without modifying the classification formats, but it does not state how macro-recall is computed when the gold set has no neutral instances. For balanced binary datasets such as Twt and Humir, standard macro-recall over the two gold classes equals accuracy, yet the table reports values like Accuracy .83 vs. Macro Recall .55 for XLM-T on Twt. These numbers are consistent only with a non-standard macro-recall that includes the absent neutral class with zero recall. The manuscript must specify the exact computation, including how classes with no gold instances are handled, and should provide the underlying contingency tables or per-instance predictions in the repository.","section":"Section 4, Table 4"},{"comment":"The 'Both: Binary' setting is ambiguous. The text says scores are recalculated by excluding 'neutral' labels, but for datasets that are already binary (Twt, Humir, TMRD-TMPR) there are no neutral gold labels to exclude. The only possible interpretation is that instances where the model predicts 'neutral' are dropped before computing scores. This protocol is not stated explicitly, is not justified, and can inflate accuracy by allowing the model to abstain on hard cases. Moreover, it is unclear whether the micro metrics are computed on the same set of remaining instances as accuracy; for example, the XLM-T/Twt row reports Accuracy .92 and Micro Recall .95, which the reader cannot verify from the information given. Please state the exact decision rule (dropping or remapping) and report the number of instances dropped per model and dataset.","section":"Section 4, Table 4 note"},{"comment":"The claim that performance 'significantly depends on the characteristics of the target text' is not supported by a controlled comparison. The four datasets differ simultaneously in source (Twitter vs. review sites), domain (movies, products, hotels), annotation scheme (emoticon-based, star ratings, manual labels), size, and time period. No analysis isolates any single characteristic, and no statistical significance test is reported. The data show that performance varies across datasets, but 'significantly depends' implies a causal or at least statistically established relation. I recommend weakening the claim to 'varies across datasets' or adding a controlled analysis—for example, stratified comparisons or a significance test—before the claim can be maintained.","section":"Abstract; Section 4; Section 5"},{"comment":"The statement that 'this study has shown that XLM-T is more effective than BERTurk' in overcoming Turkish linguistic challenges is too strong. In Table 4, XLM-T is not consistently better than BERTurk: on Humir with 'Both: Binary', BERTurk achieves .86 accuracy and .85 for XLM-T, and on Turkmenoglu with 'Both: Binary' the two are close (.81 vs. .82). The conclusion should be qualified to reflect the dataset-dependent nature of the comparison, or supported with a statistical test across datasets.","section":"Section 5, paragraph 6"}],"minor_comments":[{"comment":"The entry for Köksal and Özgür [2021] reports an accuracy of .0745 for BERTurk on BounTi; this appears to be a typo for .745 or .74 and should be corrected.","section":"Table 3"},{"comment":"The text says 'Brief descriptions, contents, and classification types of five datasets are as follows' after stating that the study focuses on four datasets; this inconsistency should be fixed, and the relationship between TMRD, TMPR, and the combined name 'TMRD-TMPR' should be clarified.","section":"Section 2.1"},{"comment":"The phrase 'Cross-Validation' in the title is misleading: the paper does not perform k-fold or any standard cross-validation procedure. If the intended meaning is cross-dataset evaluation, that term should be used and defined.","section":"Title and Section 1"},{"comment":"The column headers of Table 4 are ambiguous, with 'Recall' and 'Macro' appearing without clear ordering. The authors should explicitly list the metric sequence in the caption (e.g., Accuracy, Macro Recall, Micro Recall, Macro Precision, Micro Precision, Macro F1, Micro F1) and ensure each reported value is assigned to the correct column.","section":"Table 4"},{"comment":"The sentence 'XLM-T achieved a remarkable accuracy and F1 score of .92 and .95, respectively' does not specify whether the F1 score is macro or micro; please clarify, since the table row contains both values.","section":"Section 4, paragraph 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful resource component and the topic fits the journal, but the evaluation protocol for the key table must be made fully transparent before publication. The authors should be asked to provide the exact prediction outputs and contingency tables. The self-citation (Hürriyetoğlu et al. 2021) is contextual and does not raise a concern. If the authors cannot clarify the protocol or the numbers change materially, the central claim would need to be substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing: the survey half of this paper is a genuine contribution, and the benchmark half is not yet trustworthy. The taxonomy-labeled inventory of 31 Turkish sentiment studies and 23 datasets, with the GitHub repository, is the most useful single map of this space I've seen. The zero-shot evaluation idea is reasonable.\n\nThe stress-test note is right and it lands on Table 4. For balanced binary datasets, macro-recall is definitionally equal to accuracy, and micro-recall is always equal to accuracy. The table reports XLM-T on Twt at .83 accuracy, .83 macro-recall, and .55 micro-recall. That micro-recall cannot be standard. In the 'Both: Binary' rows, after excluding neutral predictions, accuracy, micro-recall, micro-precision, and micro-F1 are all the same quantity on the remaining instances; the table lists .92 and .95 in the same row for XLM-T on Twt. Either the columns are mislabeled, the protocol is something other than what the text says, or the numbers are simply wrong. The paper gives no way to tell, and the headline claim about text characteristics leans on these numbers.\n\nThe next soft spot is interpretive. The four datasets differ in domain, source, size, class balance, and preprocessing all at once. That configuration can't support a clean causal claim that 'characteristics' drive performance. And the comparison to best-reported numbers from the original studies is not apples-to-apples, since training setups, splits, and evaluation protocols differ.\n\nCredit where it's due: the SLR-style compilation, the taxonomy labels in Table 2, and the public repo are all useful. I would keep the survey, fix the benchmark, and re-run the metrics. This paper deserves a serious referee, but as written I would not cite the empirical numbers. If I were the editor, I'd send it out with a request for a corrected Table 4, a precise definition of the neutral-exclusion rule, and ideally per-instance predictions or runnable scripts that reproduce those exact values.","headline":"A genuinely useful map of Turkish sentiment resources, but Table 4's metrics are internally inconsistent and the headline claim is not yet supported.","tokens_in":16121,"tokens_out":7926,"would_cite":false,"duration_ms":64886,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turkish sentiment analysis tools are not portable across text types: the same model reaches .92 on one tweet set and .72 on a review set, while TSAM goes from .59 on tweets to .97 on long reviews.","keywords":["sentiment analysis","Turkish","cross-validation","taxonomy","dataset survey","XLM-T","BERTurk","TurkishBERTweet"],"falsifier":"Re-run the four models on the four datasets without dropping any predictions and compare accuracy and F1 computed over every test instance. If the .92-versus-.72 and .59-versus-.97 gaps shrink or reorder when neutral predictions count as errors, the central claim about dataset dependence is weakened; if the gaps persist with all predictions counted, the claim is supported.","tokens_in":15190,"feed_emoji":"📊","tokens_out":8387,"duration_ms":77191,"temperature":0.7,"pith_summary":"This paper tries to establish that Turkish sentiment analysis tools cannot be treated as a single reusable resource: the accuracy of a given tool depends strongly on the characteristics of the text it is applied to. To show this, the authors compiled 23 Turkish datasets used in 31 studies published between 2012 and 2022, labeled them with a taxonomy, and ran four state-of-the-art sentiment models on four recurrent datasets. The results show large swings, such as XLM-T at .92 on one tweet set but .72 on movie and product reviews, while TSAM jumps from .59 on tweets to .97 on long reviews. If true, this means reported single-dataset scores mislead practitioners, and dataset curation and preprocessing deserve as much attention as model choice.","feed_headline":"Turkish sentiment models swing from .59 to .97 by dataset","feed_subtitle":"Cross-testing four tools on four Turkish datasets shows accuracy is tied to text genre, not just model strength.","key_machinery":"The argument is carried by two coordinated devices. The first is a taxonomy of sentiment analysis studies—opinion detection type, entity, target class, data specificity, granularity, and approach—used to label the 31 collected studies and, implicitly, to describe what makes each dataset what it is. The second is a cross-dataset evaluation matrix in which each model is run on each dataset under two protocols: the native multi-class protocol and a 'Both: Binary' protocol that excludes neutral predictions. The taxonomy organizes the field; the matrix produces the evidence of domain dependence, and the binary re-scoring is what makes the swings visible, since raw multi-class scores compress all models into a narrow band.","core_discovery":"The central claim is that the performance of the sentiment analysis tools significantly depends on the characteristics of the target text. The paper tests this by applying four state-of-the-art models—XLM-T, BERTurk fine-tuned on BounTi, TSAM, and TurkishBERTweet—to four recurrent Turkish datasets and tabulating the results. Under the binary re-scoring protocol, XLM-T reaches .92 accuracy on Twt but .72 on TMRD-TMPR, while TSAM scores .59 on Twt and .97 on Humir. The authors interpret such swings as evidence that accuracy is not a fixed property of a model; source platform, label source, text length, domain, and class balance all change what a tool can do. The intended contribution is a map of Turkish sentiment analysis over 2012–2022 plus an empirical warning against treating any single dataset score as a general measure.","pith_inferences":["A standardized Turkish sentiment benchmark with fixed splits and mandatory reporting of both full-set and neutral-excluded metrics would make cross-study results comparable.","Dataset 'characteristics' could be turned into measurable features, such as platform, document length, label acquisition method, and class balance, and then used to predict how well a model will transfer to a new text type.","The same cross-dataset protocol could be applied to other morphologically rich languages, for example Arabic review-and-tweet pairs, to test whether this domain dependence is a Turkish-specific finding or a general property of sentiment tools.","The taxonomy labels could be extended with a 'source of labels' dimension—emoticon-based, rating-based, or human-annotated—because the paper's largest accuracy swings track exactly that distinction."],"forward_implications":["A model chosen for a Turkish sentiment task should be validated on text from the target platform and genre, not on a generic Turkish benchmark.","Leaderboard numbers from a single dataset cannot be compared across studies without controlling for dataset characteristics.","Classification-format mismatches, such as multi-class models applied to binary data, depress scores; resolving the mismatch by excluding neutral predictions can raise them substantially.","Modern deep models improve on classical baselines, with XLM-T's .92 on Twt compared with the .66 reported for multinomial naive Bayes, so older reported accuracies understate what is achievable.","Dataset construction choices—label source, balance, text length, and domain—are first-order variables in Turkish sentiment analysis, not minor details."],"supporting_citations":[{"why":"Supplies the taxonomy used to label the 31 studies and organize dataset characteristics.","marker":"Rodrigues et al. [2018]"},{"why":"Contributed the TMRD and TMPR datasets that serve as one evaluation corpus and reports the baselines those datasets were built on.","marker":"Demirtas and Pechenizkiy [2013]"},{"why":"Contributed the Twitter dataset that becomes the 'Turkmenoglu' evaluation set and reports an SVM baseline of .85.","marker":"Türkmenoglu and Tantug [2014]"},{"why":"Created the Twt dataset, whose .66 MNB baseline anchors the comparison on emoticon-labeled tweets.","marker":"Çoban et al. [2015]"},{"why":"Created the Humir movie and hotel review dataset where TSAM scores .97.","marker":"Ucan et al. [2016]"},{"why":"Provides the XLM-T model, a top performer in binary Turkish sentiment classification.","marker":"Barbieri et al. [2021]"},{"why":"Provides the BounTi dataset and the BERTurk model, one of the four tools evaluated.","marker":"Köksal and Özgür [2021]"},{"why":"Provides the TSAM model, whose accuracy swings most widely across datasets.","marker":"Tasar et al. [2022]"},{"why":"Provides the TurkishBERTweet model, one of the four tools evaluated.","marker":"Najafi and Varol [2023]"},{"why":"Defines the systematic literature-review procedure used to select the 31 studies.","marker":"Kitchenham and Charters [2007]"}],"fun_headline_variants":["Turkish sentiment accuracy varies by dataset: 0.59 to 0.97","Sentiment tool scores swing between .59 and .97 on Turkish data","Turkish sentiment models: accuracy depends on text type, not just model","Dataset personality drives Turkish sentiment tool performance swings","Turkish benchmark scores for sentiment tools range from .59 to .97"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 'Both: Binary' evaluation protocol is fair: it drops the model's neutral predictions before recomputing accuracy, and if those dropped instances are hard or mislabeled, the reported cross-dataset differences are artifacts of that exclusion.","fun_headline_variants_meta":{"raw":{"variants":["Turkish sentiment accuracy varies by dataset: 0.59 to 0.97","Sentiment tool scores swing between .59 and .97 on Turkish data","Turkish sentiment models: accuracy depends on text type, not just model","Dataset personality drives Turkish sentiment tool performance swings","Turkish benchmark scores for sentiment tools range from .59 to .97"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000407,"raw_usage":{"total_tokens":2080,"prompt_tokens":879,"completion_tokens":1201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1111}},"tokens_in":495,"tokens_out":1201,"duration_ms":8672,"temperature":1.0,"reasoning_tokens":1111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:08:52.844846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four models on the four datasets without dropping any predictions and compare accuracy and F1 computed over every test instance. If the .92-versus-.72 and .59-versus-.97 gaps shrink or reorder when neutral predictions count as errors, the central claim about dataset dependence is weakened; if the gaps persist with all predictions counted, the claim is supported.","supporting_citations":[],"review_version":1}