{"id":"62b44571-7f37-4805-925d-5416400167a5","arxiv_id":"2507.02137","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dataset style can be matched to platform profiles to recommend sentiment tools, and the matching heuristic is new but unvalidated.","lead":"This paper compares 14 sentiment analysis tools on 10 developer communication datasets and builds a questionnaire that recommends a tool based on a dataset's style. It gives teams a low-effort way to choose a sentiment tool for unlabeled data, though the recommendation shortcut has not yet been tested on new datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The questionnaire's tool recommendation is nearly constant: SetFit is best or near-best on every dataset in Table VII, so even perfect platform matching improves average F1 by at most 0.007 over an always-SetFit baseline, leaving the claimed improvement untested and likely vacuous.","rationale":"I read the paper in good faith. The descriptive contributions are solid: the 10-dataset, 14-tool benchmark is transparently reported, the linguistic feature coding shows high inter-rater agreement, and the platform-level differences in communication style are credible and interesting. The central claim, though, is the practical value of the questionnaire for tool selection, and that is where the soft spot is. The reader identified that the predictive link from platform similarity to tool ranking is never tested on held-out data. I agree, but I would sharpen the concern further: even granting perfect platform identification, the paper's own Table VII shows the best tool is almost always SetFit. The maximum possible average gain over an always-SetFit policy is about 0.007 overall F1, with a median of zero. That is far below any meaningful effect size and is not supported by significance testing. This is not merely a missing validation; the reported data strongly suggest the tool-recommendation component has no practical headroom. The questionnaire might still guide users toward a better tool than a weak default like SentiStrength-SE, but the paper does not define or evaluate such a baseline, so the improvement claim is unsupported. I recommend keeping the reader's conditional verdict, but the condition should explicitly require demonstrating a non-negligible improvement over a clearly specified default (e.g., always SetFit or a commonly used lexicon tool), with significance testing. If that analysis shows no improvement, the tool-selection contribution should be reframed or removed.","tokens_in":15344,"tokens_out":8997,"duration_ms":102985,"concrete_test":"Re-run the recommendation procedure using the ground-truth characteristics of the 10 evaluated datasets (i.e., perfect user answers) to obtain the questionnaire's recommended tool per dataset. Compare each recommendation's reported overall F1 from Table VII against two baselines: (a) always SetFit, and (b) the platform-specific best tool in Table VII. Report the mean and paired 95% confidence interval. If the mean improvement over always-SetFit is below 0.01 or not statistically significant, the tool-selection claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-E5 claims the mapping 'allows practical, low-barrier tool selection,' and the abstract claims dataset characteristics can 'improve tool selection.' However, the paper never compares the questionnaire's recommendation to any baseline, and its own Table VII suggests the recommendation is nearly constant. Computing from the reported overall F1 scores, SetFit is the best tool on 7 of 10 datasets (App, Code, GH1, GH3, Jira1, SO1, SO3) and is within 0.02–0.03 of the best on GH2, Jira2, and SO2. An omniscient selector that always picks the true best tool would beat an always-SetFit policy by only 0.007 average overall F1 (mean of per-dataset differences; median 0.000). No significance testing is reported, so differences of this size are within noise. The paper also never states how a platform-level 'best tool' is derived from the per-dataset results in Table VII, where datasets within a platform have different winners, and the questionnaire's 13 features include five (L5, L7, L10, L12, L13) that are uniform across all platforms (Table VIII), providing no discriminative signal. Thus the central claim that the questionnaire improves tool selection is both untested and, on the paper's own numbers, likely vacuous.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes the linguistic and statistical characteristics of 10 labeled sentiment-analysis datasets from five software-engineering platforms, evaluates 14 sentiment-analysis tools on those datasets, and uses the resulting feature and performance tables to build a questionnaire that recommends a sentiment-analysis tool for a new, unlabeled dataset. The main empirical contributions are the linguistic-feature and statistical tables (Tables V and VI), the 14×10 performance table (Table VII), and the platform-matching questionnaire (Section III-E5 and Table VIII). The paper claims that dataset characteristics can be leveraged to improve tool selection, with transformer-based models such as SetFit and RoBERTa performing strongly across datasets.","tokens_in":15614,"tokens_out":6865,"duration_ms":69677,"significance":"If the proposed mapping worked, it would be a practical, low-cost tool-selection aid for practitioners analyzing unlabeled software-engineering text, and the paper would provide a useful cross-platform benchmark of 14 sentiment tools. The dataset characterization is reasonably careful: the authors report Fleiss' kappa and raw agreement for the linguistic coding, make the labeling data available on figshare, and explicitly discuss threats to validity. However, the central claim that the questionnaire improves tool selection is not tested, and the paper's own Table VII suggests that the improvement over an always-SetFit policy would be very small, since SetFit is the best tool on 7 of 10 datasets and within 0.02–0.03 of the best on the other three. The paper also identifies future validation as needed in Section V-D, which is appropriate but leaves the headline contribution unsupported.","major_comments":[{"comment":"The central claim that the questionnaire improves tool selection is not evaluated against any baseline. The mapping and the per-platform \"best tool\" recommendations are derived from the same pooled 10-dataset evaluation, so there is no held-out or external check of whether the recommendations generalize. No confidence intervals or significance tests accompany the F1 values, even though many differences are small. Section V-D itself states that \"the questionnaire should be validated in practice to assess its utility and accuracy in real-world scenarios.\" This gap is load-bearing because the abstract and RQ3 claim that dataset characteristics improve tool selection. Concretely, from Table VII, SetFit is the best overall tool on App, Code, GH1, GH3, Jira1, SO1, and SO3, and within 0.02–0.03 of the best on GH2, Jira2, and SO2; an omniscient selector that always picks the true best tool would beat an always-SetFit policy by only 0.007 average overall F1 on these datasets. Please add a validation experiment, e.g., leave-one-dataset-out or simulated questionnaire users, comparing recommended tools against fixed defaults, and report confidence intervals or significance tests.","section":"III-E5 and IV-D"},{"comment":"The reported overall score for SentiSW on SO2 is 0.68, but the reported micro- and macro-averaged F1 are 0.79 and 0.78, whose mean is 0.785. This internal inconsistency appears in the central performance table and must be corrected, and any downstream recommendation or summary derived from that entry should be recomputed.","section":"Table VII, SO2 row"},{"comment":"The manuscript does not specify how a platform-level \"best tool\" is derived from the per-dataset results in Table VII. Within Jira, Jira1's best overall score is SetFit (0.97) while Jira2's best is ALBERT, ELECTRA, RoBERTa, or XLNet (0.84); within Stack Overflow, SO1 and SO3 prefer SetFit while SO2 prefers RoBERTa. The questionnaire recommends \"its empirically best-performing tool\" for the matched platform, but the aggregation rule (mean, median, win count, or something else) is not defined. Since RQ3's output depends directly on this rule, it must be stated and justified.","section":"III-E5c and Table VII"},{"comment":"The matching procedure weights all 13 linguistic features and 8 statistics equally, even though several inputs are non-discriminative: five features (L5, L7, L10, L12, L13) are uniform across all platforms in Table VIII, and statistics such as characters per word are nearly identical across platforms in Table VI. These constant or near-constant inputs can dilute or dominate the platform score. The Likert interval boundaries and the equal feature/statistic weights are free parameters, and no sensitivity analysis is reported. Please state which features and statistics actually separate platforms and test whether the recommendations are robust to dropping non-discriminative inputs or to changing the interval boundaries.","section":"III-E5 and Table VIII"}],"minor_comments":[{"comment":"The text says \"Every linguistic feature shows at least a two-digit range between platforms,\" but Table V reports Inquisitive with a range of only 6.8 percentage points, which is not a two-digit range; please correct this statement.","section":"V-B"},{"comment":"The URL for the scikit-learn classification_report is broken across a line break; please provide a proper citation or a working link.","section":"III-D"},{"comment":"The mapping description for \"Not specified\" and \"All\" entries is unclear: please clarify whether a \"Not specified\" answer for a feature with an \"All\" entry gives a point to every platform, to the ambiguous bucket, or to neither, and state how \"Ambiguous\" points interact with the final recommendation in a tie.","section":"III-E5 and Table VIII"},{"comment":"The conclusion states that the questionnaire \"allows users to describe their dataset and receive evidence-based tool recommendations,\" but Section V-D defers validation to future work; please qualify the conclusion to reflect that the mapping is currently a proposed, not validated, mechanism.","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"The descriptive dataset analysis and the 14-tool benchmark are useful contributions and fit the journal's scope. My main concern is that the headline contribution, the questionnaire-based tool selection, is currently unsupported and may be vacuous given the near-constant superiority of SetFit in Table VII. The authors should either add a validation study (including baseline comparisons and uncertainty quantification) or substantially reframe the contribution as a descriptive benchmark rather than a tool-selection improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is worth reading for the evaluation table: 14 tools on 10 SE sentiment datasets with micro/macro F1, plus a manual linguistic coding of 13 features across five platform types with Fleiss kappa. That is a real, transparent benchmark artifact and the inter-rater agreement is decent (substantial to almost perfect). Second, the paper's actual contribution — the questionnaire that maps dataset characteristics to a recommended tool — is not validated, and the internal evidence suggests it cannot do much. SetFit is best or within 0.02-0.03 on every dataset in Table VII. An always-SetFit policy loses to an omniscient oracle by about 0.007 average F1. So even perfect platform matching buys almost nothing, and the paper never compares the questionnaire against any baseline, never reports significance tests or confidence intervals, and never tests on held-out data. The claim in III-E5 that the mapping 'allows practical, low-barrier tool selection' is asserted, not shown.\n\nCredit where due: the feature analysis is careful. The coding procedure is described with calibration steps and raw agreement percentages, not just kappa. The statistical features are simple but sensible. The paper also honestly flags the need for practical validation in Section V-D, which the reader caught. And the finding that SE-specific lexicon tools trail transformers is consistent with prior work, so that part adds to a convergent literature rather than overturning it.\n\nSoft spots in proportion: (1) The central mapping is neither tested nor — on the paper's own numbers — likely to matter. This is the load-bearing issue. (2) There is no significance testing for the small F1 gaps that distinguish SetFit from RoBERTa, so 'best' is often within noise. (3) Table VII has an inconsistency: SO2 SentiSW overall score is 0.68 while micro/macro are 0.79/0.78, so the average would be ~0.785. That suggests a transcription or computation error somewhere. (4) Five of the 13 linguistic features (L5, L7, L10, L12, L13) are uniform across all platforms in Table VIII, so they contribute no discriminative signal; the questionnaire's point-scoring still counts them, which dilutes the match. (5) The platform-level best tool is not derived transparently: within a platform, different datasets have different winners, so the questionnaire's 'best tool for the matched platform' is not well-defined from Table VII. Also, while the raw labels are on figshare, no evaluation code is released, so reproducing Table VII would require reimplementing all 14 tool pipelines.\n\nWho is this for: researchers benchmarking SE sentiment tools, and people wanting a quick check of what has been evaluated. Practitioners should not rely on the questionnaire as-is. A serious referee should ask for a held-out validation of the mapping, baseline comparisons, confidence intervals or bootstrapping, and a fix of the SentiSW entry. This deserves peer review, but it needs major revision before the central claim can stand.","headline":"Solid descriptive benchmark; the tool-selection questionnaire is the weak link — the paper's own numbers show the recommendation barely beats always picking SetFit.","tokens_in":16131,"tokens_out":2699,"would_cite":true,"duration_ms":27130,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the best sentiment-analysis tool for a software-engineering dataset can be selected from 13 linguistic features and 8 statistics, via a questionnaire that matches the dataset to the platform whose profile it most…","keywords":["sentiment analysis","software engineering","tool selection","dataset characteristics","developer communication","questionnaire","transformer models","platform analysis"],"falsifier":"Collect a new labeled dataset from a platform or communication channel not among the five studied here, run the questionnaire's recommended tool against a fixed default such as SetFit, and check whether the recommended tool wins significantly more often than the default across several such datasets. A single held-out dataset where the recommended tool finishes below the default would directly test the predictive link.","tokens_in":15147,"feed_emoji":"💬","tokens_out":4948,"duration_ms":48706,"temperature":0.7,"pith_summary":"The paper tries to establish that the right sentiment-analysis tool for a software-engineering text dataset can be chosen from the dataset's observable characteristics. It shows that ten labeled datasets from five platforms (app reviews, code reviews, GitHub, Jira, Stack Overflow) differ measurably in 13 linguistic features and 8 statistics, and that 14 tools' F1 scores vary with those platform-linked traits. On that basis it proposes a questionnaire that maps a user's answers about a new, unlabeled dataset to the most similar platform and recommends the tool that performed best there. If it is right, researchers and practitioners get a low-barrier way to pick a trustworthy tool without labeled data or NLP expertise. The paper is an extension of prior cross-platform comparisons, aiming to turn observed variability into actionable selection guidance.","feed_headline":"Dataset traits can pick the best sentiment-analysis tool","feed_subtitle":"Answer 21 questions about your text's style and get the tool that topped the most similar platform.","key_machinery":"The load-bearing mechanism is the characteristic-to-platform mapping: 13 manually coded linguistic features (direct emotion, emphasized positivity, technical focus, balanced critique, progress sharing, gratitude, inquisitiveness, help seeking or offering, compliments, bug-fix requests, constructive criticism, username mentioning, name mentioning) plus 8 statistics (characters per document, characters per word, words per document, capitalized words, spelling mistakes, emoticons, question marks, exclamation marks). The questionnaire converts user answers about these features into interval matches (True, Likely, Unlikely, Untrue), assigns a point to each platform whose precomputed profile falls in the user's interval, uses absolute difference to match statistics, and recommends the highest-scoring platform's best tool. The mapping is what connects an unlabeled dataset to a tool recommendation without retraining or annotation.","core_discovery":"The central discovery is that platform communication styles are measurably distinct and that those distinctions carry information about which sentiment-analysis tool will do well. SetFit and RoBERTa achieve the strongest overall F1 on most of the ten datasets, yet no tool wins everywhere; the best choice tracks dataset characteristics such as direct emotion, gratitude, technical focus, and document length. The paper's proposed mapping turns these empirical regularities into a concrete procedure: interval-code the 13 linguistic features, take absolute-distance matching on the 8 statistics, count points per platform, and recommend the empirically best tool for the winning platform, falling back to SetFit or SentiStrength-SE in ambiguous cases. The authors present the mapping as a practical, low-barrier selection mechanism rather than a definitive ranking of tools.","pith_inferences":["A testable extension would be to hold out entire datasets and check whether the questionnaire's recommended tool actually beats a fixed baseline such as SetFit on the held-out data; the paper asserts this predictive link but does not report such validation.","The 13 linguistic features are hand-coded with high inter-rater agreement, so the mapping could be automated with a classifier trained on those codes, making the questionnaire objective rather than subjective.","Because class balance varies strongly across datasets, the F1-based recommendation may flip if users weight recall of negative comments more heavily; a risk-weighted variant of the mapping would be a natural next step.","The claim that platform-level similarity predicts tool ranking could be probed by a correlation analysis between pairwise characteristic distances and pairwise tool-performance differences, which the paper lists as future work."],"forward_implications":["If a user's dataset resembles a platform profile, the tool that topped that platform's benchmark should outperform a fixed default tool on the new data.","Transformer-based models (SetFit, RoBERTa, ELECTRA) become the recommended default for most developer-communication contexts, since they dominate SE-specific and lexicon tools on the evaluated datasets.","The ambiguous fallback (SetFit or SentiStrength-SE) gives a practical answer for datasets that match no platform profile, supporting robustness under dataset drift.","The questionnaire lowers the barrier to evidence-based tool selection because it requires only subjective feature answers, not labeled data or computational resources.","Platform names are a proxy rather than the target: the same machinery can be reused to match datasets directly by profile, for example via clustering."],"supporting_citations":[{"why":"Supplies the App Reviews and SO1 datasets and documents how far general sentiment tools go on SE text.","marker":"[17]"},{"why":"Provides the cross-platform evaluation setup and the emotion-to-polarity mapping heuristics used for Jira and GitHub data.","marker":"[14]"},{"why":"Documents the cross-platform tool variability and failure of tool combination that motivates the mapping approach.","marker":"[15]"},{"why":"Introduces SetFit, the model with the highest overall F1 on most datasets and a key fallback recommendation.","marker":"[40]"},{"why":"Supplies SentiStrength-SE and the Jira1 dataset, and is the dictionary-based fallback tool for ambiguous cases.","marker":"[8]"},{"why":"Supplies SentiCR and the Code Reviews dataset, anchoring the code-review platform profile.","marker":"[9]"},{"why":"Supplies the GH1 gold-standard GitHub dataset with polarity labels used for the GitHub profile.","marker":"[23]"},{"why":"Supplies the Jira2 dataset from Jira issue-tracker communication used in the Jira profile.","marker":"[26]"}],"fun_headline_variants":["Dataset style picks the top sentiment tool","No universal sentiment tool—match yours to data traits","Sentiment tool choice follows from text characteristics","Platform communication style guides tool selection","Map your data's features to a reliable sentiment analyzer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mapping assumes that the platform whose profile best matches a user's answers will also be the platform whose best-performing tool ranks highest on the user's dataset, a link the paper asserts but does not validate on held-out data.","fun_headline_variants_meta":{"raw":{"variants":["Dataset style picks the top sentiment tool","No universal sentiment tool—match yours to data traits","Sentiment tool choice follows from text characteristics","Platform communication style guides tool selection","Map your data's features to a reliable sentiment analyzer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":1990,"prompt_tokens":859,"completion_tokens":1131,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1064}},"tokens_in":475,"tokens_out":1131,"duration_ms":13503,"temperature":1.0,"reasoning_tokens":1064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:36:34.002956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a new labeled dataset from a platform or communication channel not among the five studied here, run the questionnaire's recommended tool against a fixed default such as SetFit, and check whether the recommended tool wins significantly more often than the default across several such datasets. A single held-out dataset where the recommended tool finishes below the default would directly test the predictive link.","supporting_citations":[{"cited_title":"Sentiment analysis for software engineering: How far can we go?","cited_arxiv_id":null,"evidence_quote":"Supplies the App Reviews and SO1 datasets and documents how far general sentiment tools go on SE text."},{"cited_title":"Can we use se-specific sentiment analysis tools in a cross-platform setting?","cited_arxiv_id":null,"evidence_quote":"Provides the cross-platform evaluation setup and the emotion-to-polarity mapping heuristics used for Jira and GitHub data."},{"cited_title":"Efficient few-shot learning without prompts,","cited_arxiv_id":null,"evidence_quote":"Introduces SetFit, the model with the highest overall F1 on most datasets and a key fallback recommendation."},{"cited_title":"Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,","cited_arxiv_id":null,"evidence_quote":"Supplies SentiStrength-SE and the Jira1 dataset, and is the dictionary-based fallback tool for ambiguous cases."},{"cited_title":"Senticr: A customized sentiment analysis tool for code review interactions,","cited_arxiv_id":null,"evidence_quote":"Supplies SentiCR and the Code Reviews dataset, anchoring the code-review platform profile."},{"cited_title":"A gold standard for polarity of emotions of software developers in github,","cited_arxiv_id":null,"evidence_quote":"Supplies the GH1 gold-standard GitHub dataset with polarity labels used for the GitHub profile."},{"cited_title":"The emotional side of software developers in jira,","cited_arxiv_id":null,"evidence_quote":"Supplies the Jira2 dataset from Jira issue-tracker communication used in the Jira profile."}],"review_version":1}