{"id":"73e8607e-cfdc-4dd7-955f-e701b8852b83","arxiv_id":"2507.20249","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper claims linguistic features such as discourse markers and prefaces can both predict perceived professionalism and distinguish human from LLM-generated expert questions, but the correlation evidence is internally inconsistent and the professionalism construct is never directly modeled.","lead":"The paper proposes that structural and pragmatic language features can model professionalism in expert questioning, tested on financial earnings call questions. It reports a feature-based classifier that beats Gemini and SVM at telling human from AI-written questions, but the paper's own correlation tables undercut the claim that the same features consistently signal both professionalism and human authorship.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central convergence claim is contradicted by its own Table 3: several features flip correlation direction between the professionalism and origin tasks.","rationale":"The reader's formal weakest assumption was annotation reliability, but the reader's rationale also flagged the sign flips. My concern is more direct and more damaging: the paper's own correlation table contradicts the claimed convergence, independent of annotation quality. Even with perfect annotations, the reported directional reversals (open questions, acknowledgment, interjection count) invalidate the central inference that origin is a valid proxy for professionalism. This is an internal inconsistency, not a mere missing control. A bootstrap or matched-item analysis could in principle salvage the claim if the flips are noise, but as presented the evidence is self-contradictory. Therefore the verdict should remain REJECT: the central claim is not merely under-supported; it is contradicted by data in the paper. I agree with the reader's bottom line, though I locate the load-bearing flaw in the internal sign flips rather than in annotation reliability.","tokens_in":126,"tokens_out":1525,"duration_ms":33581,"concrete_test":"On the Human-Rated Professionalism Dataset (HRPD), compute Spearman correlations between each linguistic feature and the binary human/LLM origin label for the same 250 questions (or the subset with both labels), and compare these to the reported feature–professionalism correlations. If features such as open questions, acknowledgment, and interjection count show opposite signs between origin and professionalism on the identical items, then the proxy assumption fails directly. Additionally, compute a bootstrap 95% confidence interval for each QOD correlation to check whether the sign flips in Table 3 are statistically robust or within noise.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.3 asserts that 'many of the same linguistic indicators ... exhibit consistent correlation direction and strength in both tasks,' and this convergence is the paper's sole justification for treating question origin as a proxy for professionalism. However, the paper's own Table 3 shows multiple features with opposite signs between HRPD (professionalism ratings) and QOD (origin): open questions are significantly negative in HRPD (↓↓↓) but significantly positive in QOD (↑); acknowledgment is strongly positive in HRPD (↑↑↑) but strongly negative in QOD (↓↓↓); interjection count is strongly positive in HRPD (↑↑↑) but strongly negative in QOD (↓↓↓). These are not minor discrepancies—they are outright directional reversals on load-bearing features. If the same features push toward professionalism but away from human authorship, then the origin-based classifier's high accuracy (0.96) cannot be interpreted as evidence that it captures professionalism; it may simply be exploiting authorship-specific artifacts (e.g., LLM stylistic tics) that are uncorrelated or inversely correlated with perceived professionalism. The paper never reports the correlation between the two target variables on matched items, nor does it test whether the feature-origin correlations align with feature-professionalism correlations beyond a selective reading of Table 3. Thus the central claim—that origin is a valid, linguistically grounded proxy for professionalism—is not supported; in fact, the reported data undermine it. This is an internal inconsistency in the paper's evidence, not a dispute with outside consensus, and it cannot be fixed by appealing to 'consistent direction and strength' when the appendix contradicts that statement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to model professionalism in expert questioning by extracting hand-crafted linguistic features from financial analyst questions and correlating them with two targets: human-perceived professionalism ratings (HRPD) and question origin (human vs. LLM, QOD). The authors report that many features show consistent correlations with both targets, argue that question origin is therefore a valid proxy for professionalism, and train a Random Forest classifier on the origin task that reportedly reaches 0.96 accuracy, outperforming an SVM and an LLM baseline. The paper concludes that professionalism is learnable from interpretable linguistic signals.","tokens_in":5811,"tokens_out":4587,"duration_ms":56768,"significance":"If the convergence claim were sound, this would be a valuable contribution: it would offer an interpretable, theory-driven way to evaluate professional questioning and to detect LLM-generated content in high-stakes domains. The paper also provides a potentially reusable feature scheme and a publicly available supplementary dataset, which are constructive elements. However, the central empirical claim is contradicted by the paper's own Table 3, which shows multiple statistically significant sign reversals between the professionalism and origin correlations. Because the proxy argument rests entirely on these correlations aligning, the main thesis is not supported. The classification results are also insufficiently specified. The datasets and feature inventory may still be useful to the community, but the current analysis does not establish the paper's core conclusion.","major_comments":[{"comment":"The central convergence claim is directly contradicted by the paper's own data. The text states that 'many of the same linguistic indicators ... exhibit consistent correlation direction and strength in both tasks,' but Table 3 shows multiple statistically significant directional reversals: acknowledgment is ↑↑↑ in HRPD and ↓↓↓ in QOD; open question type is ↓↓↓ in HRPD and ↑ in QOD; interjection count is ↑↑↑ in HRPD and ↓↓↓ in QOD. These are not minor discrepancies but sign flips on features the paper uses to argue for a shared stylistic foundation. Since the proxy argument in §3.3 rests on alignment between the two correlation profiles, the presence of these reversals means the claim of convergence is unsupported. The paper provides no aggregate measure, such as a rank correlation across feature vectors, to quantify the extent of agreement, so the selective reading of Table 3 is insufficient.","section":"§3.3, Table 3"},{"comment":"The proxy logic is structurally circular and is never directly tested. The paper establishes that features correlate with professionalism (HRPD) and with origin (QOD) in separate datasets, then treats origin classification accuracy as evidence that the features capture professionalism. This requires the unstated assumption that the two target variables are positively associated on matched items, which is never measured. The reported 0.96 accuracy could instead reflect LLM-specific artifacts that are unrelated or inversely related to perceived professionalism; indeed, interjection count is strongly positive for professionalism but strongly negative for human origin in Table 3. A direct test—for example, training on HRPD professionalism scores, or measuring whether predicted origin correlates with professionalism ratings on held-out items—is needed to support the paper's central claim.","section":"§3.3, §4"},{"comment":"The feature annotation process is undocumented, making the reported correlations unverifiable. The paper does not provide annotation guidelines, annotator training, inter-annotator agreement statistics, or a description of whether features were extracted manually or automatically. Since every correlation in Table 3 and the classifier in Section 4 rely on these features, the absence of reliability evidence undermines the empirical foundation. This concern is amplified because HRPD and QOD are annotated separately, and any difference in annotation practice between the two datasets could produce spurious sign flips or false alignments.","section":"§3.1, §3.2"},{"comment":"The classification evaluation is insufficiently specified. The paper does not report the train/test split, whether cross-validation was used, which dataset(s) the results refer to, how the 'full question set' was partitioned, or the hyperparameters of the Random Forest and SVM. Without this information, the accuracy and F1 values in Table 2 cannot be interpreted, and the comparison to gemini-2.0-flash may not be controlled (e.g., different prompting strategies, different training-data exposure, or label leakage from HRPD items that also have origin labels). The claim that a shallow feature-based model outperforms an LLM requires a fully specified evaluation protocol.","section":"§4, Table 2"}],"minor_comments":[{"comment":"The caption of Table 1 says it presents all statistically significant correlations in the 'mixed and combined datasets,' but the manuscript never defines what 'mixed' and 'combined' mean; also, Table 1 is a strict subset of Table 3, yet both are captioned as reporting significant correlations, which is confusing.","section":"§3.2"},{"comment":"The feature taxonomy in the text lists four categories (discourse regulators, prefaces, question types, request types), but Table 3 also includes NLP features (type-token ratio, readability, word/sentence counts) and additional features such as filler word count, assertion count, and mean assertion length that are not described in Section 3.2.","section":"§3.2, Table 3"},{"comment":"The phrase 'full question set' is ambiguous; the authors should specify whether it consists of QOD only, HRPD only, or both, and how the origin label was obtained for HRPD items.","section":"§4"},{"comment":"The provenance of HRPD is incompletely described: the text says it comes from Juan et al. (2025) and was shared upon request, while the supplementary link is for QOD; the authors should clarify how readers can access HRPD and confirm that its annotation procedure matches the description given.","section":"§3.1, Supplementary materials"},{"comment":"The tables report Spearman correlations but no sample sizes are shown; the authors should report N for each correlation to allow readers to assess the precision of the estimates.","section":"Tables 1 and 3"}],"recommendation":"reject","confidential_remarks":"The reader's report and the stress-test note are correct: the paper's own Table 3 defeats its central convergence claim. I do not see a straightforward fix within the current scope, because the proxy argument is load-bearing and the empirical evidence contradicts it. A future version that drops the professionalism-proxy framing and instead positions itself as a purely descriptive corpus study, or that directly models professionalism with the HRPD ratings, could be reconsidered. The self-citations and co-authored dataset are not problematic per se, but the reliance on a prior paper's annotations without any reliability information is a concern for this journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know: this is a short empirical paper with a genuinely new annotated dataset and a clean feature-based classifier, but the reader's stress test lands. The central convergence claim is contradicted by the paper's own Table 3, and that contradiction is load-bearing, not cosmetic.\n\nCredit where it is due. The Question Origin Dataset (250 human analyst questions paired against 250 LLM-generated ones) is new and appears to be available. The Random Forest result, hitting 0.96 accuracy on interpretable features and beating both a TF-IDF SVM and Gemini, is a useful, reproducible baseline for detection of machine-written questions in financial text. The feature set is grounded in prior discourse work, and the paper's limitations paragraph is honest about the single-domain scope and about origin being an imperfect proxy.\n\nThe soft spot is the center of the argument. Section 3.3 says the same linguistic indicators show consistent correlation direction and strength in both the professionalism and origin tasks. That is not what the data show. Acknowledgment is strongly positive for professionalism and strongly negative for human origin; open questions flip from strongly negative to positive; interjection count flips from strongly positive to strongly negative. The paper never acknowledges these reversals, and it never tests the proxy directly by correlating origin labels with professionalism ratings on matched items. This matters because the 0.96 accuracy could be driven by authorship-specific artifacts that have nothing to do with perceived professionalism. Also, classification results have no error bars or significance tests, and the annotation section gives no guidelines and no inter-annotator agreement, so we cannot tell whether the features were coded consistently across the two datasets.\n\nWho is this for? Financial NLP researchers who want a cheap, transparent detector of LLM-generated analyst questions, and computational pragmatics people interested in operationalizing question style. The paper deserves a serious referee: the dataset and code make it worth engaging, and the convergence claim is testable. I would send it to review with a clear request to either test the proxy directly or substantially weaken the claim to something like \"some features transfer, others do not, and here is the pattern.\"","headline":"The dataset and the shallow classifier are real contributions, but the paper's central proxy claim is not supported by its own correlation table.","tokens_in":6381,"tokens_out":2935,"would_cite":false,"duration_ms":39689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that professionalism in expert questioning is a learnable, interpretable linguistic pattern that can be modeled with a small set of structural and pragmatic features, and that question origin serves as a valid proxy for it.","keywords":["professionalism","expert questioning","linguistic features","earnings calls","question origin classification","interpretable machine learning","LLM-generated text detection","discourse pragmatics"],"falsifier":"Collect independent professionalism ratings for the 250 questions in the Question Origin Dataset, then recompute the Spearman correlations between features and ratings; if the correlation directions do not match the paper's Human-Rated Professionalism Dataset columns, the claim that origin is a valid proxy for professionalism fails. A second test would re-annotate a sample with two annotators and report agreement; low agreement would show the feature definitions are not stable enough to carry the result.","tokens_in":5355,"feed_emoji":"🎙️","tokens_out":5942,"duration_ms":63836,"temperature":0.7,"pith_summary":"This paper argues that professionalism in expert questioning is not an elusive quality but a measurable linguistic pattern. It introduces an annotation scheme for structural and pragmatic features of financial analyst questions—discourse regulators, prefaces, question types, and request types—and shows that these features correlate in the same direction and strength with both human-rated professionalism and with whether a question was written by an analyst or generated by an LLM. The paper claims this convergence makes question origin a valid proxy for professionalism. It also finds that professional questions tend to be shorter, more readable, and less cluttered with prefatory framing, and that a Random Forest trained only on these features reaches 0.96 accuracy in distinguishing expert from LLM questions, beating both an SVM on raw text and gemini-2.0-flash.","feed_headline":"Shallow language features beat Gemini at spotting expert questions","feed_subtitle":"A random forest using prefaces, request types, and readability hits 96 percent accuracy on human vs. LLM questions.","key_machinery":"The load-bearing instrument is a hand-annotation framework that maps each question onto four linguistically motivated categories: discourse regulators (elements like 'Thanks' or 'this one is for you' that manage the interaction), prefaces (propositions before the question that justify or contextualize it), question types (open, polar, closed-list), and request types (explanation, clarification, confirmation, data, opinion), together with readability and lexical metrics such as type-token ratio, Flesch-Kincaid, and Dale-Chall. The machinery does its work by turning the 'how' of a question into a fixed feature vector, so that correlations with human professionalism ratings and with question origin can be computed and a Random Forest can be trained on the same features. Because the features are defined in advance and are inspectable, the framework also supplies an explanation of what the classifier has learned: prefatory padding and low readability mark machine-style questions, while concise, thematically focused, readable questions read as professional.","core_discovery":"The central claim is that the same set of interpretable linguistic indicators carries both human perceptions of professionalism and the distinction between expert-authored and machine-generated questions. Refined, this claim says that a question's professionalism is largely encoded in its surface-structural and pragmatic design: how it acknowledges the prior speaker, how it frames or prefaces the request, which syntactic question form it uses, what kind of answer it solicits, and how readable it is. The paper reports that these features align consistently across the Human-Rated Professionalism Dataset and the Question Origin Dataset, and interprets the alignment as evidence that origin classification can stand in for professionalism when human ratings are unavailable. A corollary finding is that clarity, not complexity, drives perceived professionalism: higher ratings go with fewer words and sentences and higher readability scores. Finally, the paper shows that a shallow classifier using only these features outperforms a large language model and a bag-of-words SVM on the origin task, which it presents as evidence that professionalism is learnable and expressible in transparent linguistic terms.","pith_inferences":["If the convergence holds beyond finance, the origin task could become a cheap labeling device for professionalism in other expert question settings such as medical or legal Q&A; the paper does not test that transfer.","A natural ablation would separate feature groups (prefaces vs. question types vs. readability) to check whether the 0.96 accuracy is carried by a single superficial cue such as word count; the paper does not report such an ablation.","The professionalism ratings may partly encode rater expectations about what an analyst should sound like, so the features' correlation with origin could be inflated by the LLM's distinct style; a test with a different LLM family or with human imitations would separate stylistic origin signals from professionalism itself.","Because the paper's Question Origin Dataset labels are binary, it treats professionalism as a category rather than a gradient; a rated version of that dataset would show whether the proxy saturates once questions are genuinely professional."],"forward_implications":["Professionalism in financial analyst questions can be scored automatically with transparent, pre-defined linguistic features instead of expensive human annotation.","The same features give an interpretable diagnostic: a question flagged as machine-generated can be shown to be over-prefaced, low-readability, or low on discourse regulators.","Origin classification is a viable proxy for professionalism, so large unlabeled pools of expert and LLM-generated questions can be converted into training signal without collecting ratings.","Question-generation systems that aim to sound professional should reduce prefatory framing and aim for concise, readable constructions, since those are the traits associated with higher human ratings."],"supporting_citations":[{"why":"Supplies the five-annotator professionalism ratings in the Human-Rated Professionalism Dataset, the target for the correlation analysis.","marker":"Juan et al. (2025)"},{"why":"Provides corpus evidence that LLM-generated and human texts differ systematically, motivating the origin-based contrast.","marker":"Berber Sardinha (2024)"},{"why":"Documents human variability versus machine consistency in text, the basis for treating origin as a stable stylistic signal.","marker":"Zanotto and Aroyehun (2024)"},{"why":"Establishes earnings calls as a distinct financial reporting genre, fixing the domain of the datasets.","marker":"Crawford Camiciottoli (2010)"},{"why":"Characterizes analyst questioning strategies in earnings conference calls, grounding the request-type and preface features.","marker":"de Oliveira and Pereira (2018)"},{"why":"Defines institutional talk as role-shaped and goal-oriented, the theoretical basis for treating professionalism as measurable.","marker":"Drew and Heritage (1992)"},{"why":"Shows how question form (yes/no vs. wh-) signals stance and control, motivating the question-type feature.","marker":"Clayman and Heritage (2002)"},{"why":"Prior computational model of analysts' questioning strategies that this work extends toward professionalism.","marker":"D'Agostino et al. (2024)"}],"fun_headline_variants":["Linguistic tells expose AI-generated expert questions","Surface style outs LLM-authored financial queries","Professional tone is just a few language cues","Interpretable features beat deep models on expert Qs","Clear, simple questions read as more professional"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-annotated linguistic features were marked accurately and with the same standards in both datasets; the paper gives no annotation guidelines and no inter-annotator agreement scores, so inconsistent or noisy feature labels would undermine every correlation and the reported 0.96 classifier accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Linguistic tells expose AI-generated expert questions","Surface style outs LLM-authored financial queries","Professional tone is just a few language cues","Interpretable features beat deep models on expert Qs","Clear, simple questions read as more professional"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1264,"prompt_tokens":899,"completion_tokens":365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":515,"tokens_out":365,"duration_ms":4925,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:41:41.459612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect independent professionalism ratings for the 250 questions in the Question Origin Dataset, then recompute the Spearman correlations between features and ratings; if the correlation directions do not match the paper's Human-Rated Professionalism Dataset columns, the claim that origin is a valid proxy for professionalism fails. A second test would re-annotate a sample with two annotators and report agreement; low agreement would show the feature definitions are not stable enough to carry the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the five-annotator professionalism ratings in the Human-Rated Professionalism Dataset, the target for the correlation analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides corpus evidence that LLM-generated and human texts differ systematically, motivating the origin-based contrast."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes earnings calls as a distinct financial reporting genre, fixing the domain of the datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines institutional talk as role-shaped and goal-oriented, the theoretical basis for treating professionalism as measurable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior computational model of analysts' questioning strategies that this work extends toward professionalism."}],"review_version":1}