{"id":"25c7e90a-8ae7-4a1b-a3a9-373f2bb465e1","arxiv_id":"2507.09565","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Holistix is a 1420-post Australian mental-health forum dataset annotated for six wellness dimensions, benchmarked with classical ML and transformer classifiers.","lead":"A new dataset, Holistix, labels 1420 posts from an Australian mental health forum with six wellness dimensions and explanatory text spans. The paper benchmarks classical and transformer classifiers on it, with MentalBERT reaching 74% accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset gold-standard labeling pipeline is underspecified: no final-label adjudication rule for the two student annotators, only a single overall kappa, and admitted Emotional/Spiritual subjectivity make Table IV per-class F1s uninterpretable until per-class reliability is established.","rationale":"The reader's conditional verdict already identifies annotation reliability as the weakest assumption, and I agree. I deliberately prioritize this over the fixed-split/cross-validation contradiction because the labeling pipeline threatens the validity of the dataset itself, whereas the evaluation-protocol issue only affects the reported accuracy number and can be corrected by a clarifying statement. The paper has independent support in the form of a public GitHub release and a detailed annotation framework, so the concern is not about absence of effort but about an underspecified and internally inconsistent reliability check. The proposed concrete test is cheap and decisive: inspect the repository for per-item annotator labels and adjudication rules, then recompute per-class agreement. If the test shows robust double labeling with explicit resolution of disagreements, the conditional can be lifted; if not, the benchmark cannot yet be treated as a trustworthy gold standard and the per-class F1 values in Table IV should not be interpreted as meaningful measures of classifier capability.","tokens_in":9753,"tokens_out":4901,"duration_ms":57883,"concrete_test":"Inspect the released GitHub repository for the actual annotation artifacts. Specifically: (1) count how many of the 1420 posts carry two independent annotator labels versus a single label; (2) look for any adjudication log, final-label decision rule, or revised-guideline document; (3) recompute per-class inter-annotator agreement (Cohen's kappa or Fleiss' kappa with appropriate rater count) on the double-labeled subset. If fewer than ~80% of posts are double-labeled, or if the per-class kappa for EA or SpiA is below 0.4, the gold-standard reliability and all downstream F1 numbers are not supported. If full double labeling and a documented adjudication pipeline exist, re-run the MentalBERT 10-fold evaluation on the adjudicated labels to confirm the Table IV numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is a labeled benchmark, so the validity of the gold labels is load-bearing. Section II-E says two student annotators independently annotated the data and Fleiss' kappa was 75.92%, but the paper never states whether both annotators labeled all 1420 posts or whether a second annotator only spot-checked 20% (as suggested by guideline 7 in Section II-D.1). No adjudication step is described for cases where the two annotators disagreed, yet the released dataset presumably contains one gold label per post. This is not a minor reporting gap: the absence of per-class agreement is decisive because Section IV itself concedes that Emotional and Spiritual categories are subjective and produced divergent interpretations of the same text (e.g., 'I don't belong anywhere' labeled Social vs Emotional). If the final labels for the majority of posts reflect a single student annotator's judgment, or if disagreements were resolved arbitrarily, then every result in Table IV—notably MentalBERT's EA F1 of 0.48 and SpiA F1 of 0.52—measures classifier performance against a noisy, unverified target. Because the paper's central claim is the creation of a reliable benchmarking resource, this unresolved labeling pipeline is the most load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Holistix, a dataset of 1,420 posts from Australia's Beyond Blue mental health forums, each annotated with one of six wellness dimensions (physical, emotional, social, intellectual, spiritual, vocational) together with explanatory text spans. The authors describe an expert-developed annotation framework and report an overall Fleiss' kappa of 75.92%. They then evaluate classical machine-learning baselines (logistic regression, linear SVM, Gaussian NB) and fine-tuned transformer models (BERT, DistilBERT, MentalBERT, Flan-T5, XLNet, GPT-2) for single-label multi-class classification, with MentalBERT achieving the highest reported accuracy of 74%. The paper also applies LIME to MentalBERT and logistic regression and reports ROUGE, BLEU, precision, recall, and F1 scores for the resulting explanations against the annotated spans. The central claims are that Holistix is a reliable, reusable wellness-dimension benchmark for the Australian population and that the reported baselines establish a meaningful performance landscape.","tokens_in":10003,"tokens_out":5857,"duration_ms":67933,"significance":"If the gold labels are trustworthy, Holistix would fill a concrete gap: it is a publicly released, expert-grounded dataset covering all six wellness dimensions in a region-specific population, extending prior four-dimension resources such as Wellxplain. The paper provides a clear taxonomy, worked indicator examples, and open code and data on GitHub, which are good signs for reproducibility. However, the value of the benchmark hinges on the annotation pipeline: two load-bearing gaps—the unresolved final-label adjudication rule and the absence of per-class agreement—mean that the reported per-class F1 scores cannot yet be interpreted as measurements against a reliable gold standard. The evaluation-protocol ambiguity and missing statistical uncertainty further weaken the model-comparison claims. These issues are fixable within the manuscript's scope, but they must be resolved before the dataset can serve as a dependable benchmarking resource.","major_comments":[{"comment":"The manuscript states: 'We used a fixed set of 990 training samples, 212 validation samples, and 213 test samples, with performance evaluated using 10-fold cross-validation.' A fixed split and 10-fold cross-validation are mutually incompatible evaluation protocols, yet Table IV is captioned as averaged over 10 folds and the abstract repeats the 10-fold claim. Please specify which protocol produced the reported numbers, how the validation set is used during hyperparameter selection, and how the 1,420 posts are partitioned across folds. Without this clarification, the headline 74% MentalBERT accuracy and the per-class F1 scores are not reproducible.","section":"Section III, first paragraph; Table IV"},{"comment":"The final-label pipeline is underspecified. Section II-E says two student annotators 'annotate the data instances independently,' but guideline 7 in Section II-D.1 mentions only that quality can be checked by having a second annotator 'review 20% of the entries.' The paper never states whether both annotators labeled all 1,420 posts or whether the second annotator reviewed only a subset, and no adjudication step is described for converting the annotators' labels into the single gold label per post implied by Table II. Because Section IV itself concedes that emotional and spiritual dimensions produce divergent interpretations of the same text, an explicit rule for resolving disagreements and per-class inter-annotator agreement are needed before the Table IV per-class F1 values can be treated as measuring a known-good target.","section":"Section II-E and Section II-D.1, guideline 7"},{"comment":"All classification results are reported as point estimates with no standard deviations, confidence intervals, or significance tests. With per-class sample sizes between 150 and 406, differences such as MentalBERT's Vocational F1 of 0.87 versus DistilBERT's 0.79, or BERT's Emotional F1 of 0.39 versus MentalBERT's 0.48, may well be within fold-to-fold noise. The paper should report per-fold variance and, ideally, paired significance tests before claiming that MentalBERT 'is the top choice' and that DistilBERT and GPT-2 offer 'strong performance' on the basis of these numbers.","section":"Table IV and Section III-B"},{"comment":"The LIME-span evaluation is too underspecified to interpret. The sentence 'calculate the similarity score between the LIME-generated predictions and the annotated explanation spans using keywords' does not say how LIME's feature-level explanations are aligned with the annotated spans, how keywords are matched (exact tokens, stemming, partial overlap), or what the units of comparison are. ROUGE and BLEU are usually defined for generated text versus a reference, so the exact input format must be described. As written, the claim that MentalBERT outperforms logistic regression across all interpretability metrics is not reproducible.","section":"Section III-B, Explainability Evaluation; Table V"}],"minor_comments":[{"comment":"The term 'Perplexity' appears to be used where 'ambiguity' is meant; please define the term explicitly or replace it.","section":"Section II-D heading"},{"comment":"The column header 'Avg. Count' appears to list raw frequencies (e.g., job(45)), not average counts; please clarify or relabel the column.","section":"Table III"},{"comment":"For reproducibility, report the transformer fine-tuning details: optimizer, warmup steps, maximum sequence length, number of runs, and early-stopping criterion, and explain why the BERT-family learning rate is set to 1e-3.","section":"Section III-A"},{"comment":"The interpretation that the lower counts of IA and VA 'may indicate that these areas are well-regulated and efficiently managed' is an unsupported causal claim; please soften it or provide supporting evidence.","section":"Section II-C"},{"comment":"Please specify the date range of the scraped posts, the exact selection criteria for threads and subforums, and whether the scraping complied with Beyond Blue's terms of service.","section":"Section II-A"},{"comment":"Since two annotators are used, please justify the choice of Fleiss' kappa over Cohen's kappa, or clarify that the reported value is equivalent to Cohen's kappa for the two-rater case.","section":"Section II-E"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worthwhile and the dataset fills a real niche, but the annotation-pipeline ambiguity is exactly the kind of issue that determines whether the released resource is usable by the community. I would ask the authors to provide the annotation logs or at least a clear statement of how many posts received double annotation and how disagreements were resolved, along with per-class agreement. The fixed-split versus 10-fold contradiction also needs to be resolved in the revision, as it affects every number in Table IV."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is the contribution, and it's a reasonable one; the paper around it is a standard baseline-and-dataset report, but the labeling pipeline and evaluation protocol have gaps that need closing before the benchmark numbers carry weight. If those get fixed, this is a solid resource for region-specific mental health NLP.\n\nWhat's new: Holistix is 1420 posts from Australia's Beyond Blue forums, annotated for six wellness dimensions (physical, emotional, social, intellectual, spiritual, vocational) with explanatory text spans. That extends Wellxplain's four-dimension Reddit scheme to a new platform, six classes, and a specific population. The annotation framework follows Dunn/Hettler and includes expert-developed guidelines; the dataset is released on GitHub. That is a real contribution, and the paper's baseline comparison (traditional ML plus six transformers) is a reasonable first benchmark. The reported 76% kappa between two student annotators, even if coarse, is decent.\n\nThe soft spots are mostly about transparency. The biggest one, correctly flagged in the stress-test, is the gold-label construction. Section II-E says two student annotators independently annotated the data and reports one overall Fleiss kappa. It never states whether both annotated all 1420 posts or one did full and the other spot-checked 20% (guideline 7 in II-D.1). There is no adjudication step for disagreements, yet the released dataset has one label per post. Given that the authors themselves concede that Emotional and Spiritual are subjective and subject to divergent readings, per-class agreement and a described resolution rule are not a nicety—they're needed to interpret Table IV, where EA and SpiA are exactly the classes with low F1. This isn't fatal, but it's load-bearing.\n\nAlso, the evaluation protocol is ambiguous: abstract and Table IV say 10-fold CV, but Section III describes a fixed 990/212/213 split. No error bars or variance over folds, so the ranking of models is unquantified. The LIME similarity comparison is underspecified (how LIME-selected features are aligned with human spans). Minor: Fleiss' kappa is designed for >2 raters; Cohen's kappa is standard for two.\n\nWho's this for: anyone building or using wellness-dimension datasets from social media in the mental-health NLP subfield, especially for region/culture-specific resources. It deserves peer review: the resource is new, potentially reusable, and the flaws are fixable reporting gaps rather than a bad central idea. Send it out, but the reviewers should insist on clarification of the annotation pipeline and per-class reliability, plus a consistent evaluation description.","headline":"A useful new wellness-dimension dataset for mental-health NLP, but the gold-label pipeline and evaluation protocol need to be spelled out before the benchmark numbers are trusted.","tokens_in":10508,"tokens_out":2618,"would_cite":false,"duration_ms":29616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Holistix introduces 1,420 forum posts labeled for six wellness dimensions, with the best model reaching 74% accuracy.","keywords":["wellness dimensions classification","mental health narratives","social media dataset","Beyond Blue forum","text span annotation","multi-class classification","MentalBERT","explainability"],"falsifier":"Re-annotate the 223 emotional posts and a sample of the 190 spiritual posts with a fresh group of clinical psychologists using the same published guidelines; if their labels agree with the released labels only at chance levels (for example, kappa below 0.4), or if the disagreement concentrates on the same ambiguous phrases flagged in the limitations, then the reported F1 scores for those dimensions measure label noise rather than model skill.","tokens_in":9568,"feed_emoji":"🧠","tokens_out":3453,"duration_ms":35168,"temperature":0.7,"pith_summary":"The paper introduces Holistix, a dataset of 1,420 posts from Australia's Beyond Blue mental health forums, each labeled by trained annotators into one of six wellness dimensions: physical, emotional, social, intellectual, spiritual, and vocational, together with the text spans that justify the label. The aim is to make mental-health assessment from social media more fine-grained: instead of predicting a binary crisis or not, a model can identify which dimension of a person's life is under strain. The paper reports that a fine-tuned MentalBERT model reaches 74% accuracy over ten-fold cross-validation, with strong F1 scores for vocational, social, and physical categories and much weaker ones for emotional and spiritual categories. It also evaluates LIME explanations against the annotated spans, connecting the classification to a transparent, human-readable justification. A sympathetic reading is that the dataset, released publicly, provides a region-specific resource for moving from crude mental-health signals toward dimension-level, explainable assessment.","feed_headline":"New dataset maps mental-health posts to six wellness dimensions","feed_subtitle":"1420 Beyond Blue forum posts, expert-labeled with explanatory spans; best model hits 74% accuracy.","key_machinery":"The load-bearing object is the annotation framework: a six-class label scheme derived from Dunn's theory of high-level wellness and Hettler's Six Dimensions model, operationalized through a table of class indicators (for example, fatigue and sleep for physical, career dissatisfaction for vocational, loneliness for social) and a set of perplexity guidelines for resolving overlap, ambiguity, and metaphor. Two student annotators applied this framework independently, with a Fleiss kappa of 0.7592, and the expert-developed guidelines supplied the indicators and resolved ambiguous cases. This framework does the work of turning raw forum text into a supervised learning signal; the classifier scores and the explanation-span evaluation are both measured against it.","core_discovery":"The paper's central claim is that wellness-dimension classification with explanatory span annotation is a viable, finer-grained task for mental-health text, and that the Holistix dataset supports it. On its own terms, the contribution is the dataset itself: 1,420 preprocessed posts from Beyond Blue forums, annotated under expert-designed guidelines grounded in Dunn and Hettler's six-dimension model, with each post assigned a dominant wellness aspect and key text spans highlighted. The evaluation establishes that transformer models fine-tuned on this dataset clearly outperform classical TF-IDF classifiers (74% versus 52% best accuracy), that MentalBERT is the strongest baseline, and that emotional and spiritual aspects are the hardest classes across all models, with MentalBERT's emotional F1 at 0.48. The paper further claims that LIME-generated explanations align with the human span annotations, with MentalBERT ahead of logistic regression on F1, ROUGE, and BLEU, showing that the span labels are not just decoration but a measurable explanation target.","pith_inferences":["The single-label design may undercount the co-occurring dimensions that the perplexity guidelines explicitly acknowledge, so a multi-label version of the same corpus could yield higher measured performance and better reflect the reality of mental-health narratives.","The difficulty with emotional and spiritual classes may be an annotation-subjectivity problem as much as a model deficiency; treating those labels as soft or uncertainty-weighted could be more informative than forcing a single dominant class.","Because the data come from one Australian forum, models trained on Holistix may not transfer to other regions or platforms without fine-tuning, making the dataset a natural test bed for cross-cultural mental-health NLP adaptation.","The annotated spans could be used to train extractive explanation models directly, rather than only evaluating post-hoc LIME output, which would turn explanation generation into a supervised task on this resource."],"forward_implications":["The public Holistix dataset gives researchers a region-specific benchmark for classifying wellness dimensions in social-media mental-health text, with explanatory spans that go beyond simple class labels.","Since transformer models clearly beat classical TF-IDF classifiers, future work on this task can reasonably focus on pretrained or instruction-tuned language models rather than feature engineering.","The consistently low F1 scores for emotional and spiritual dimensions across all models indicate that these two classes need either more training data, different label definitions, or alternative representations before they are reliable enough for deployment.","The span annotations backed by LIME similarity metrics provide a concrete way to evaluate whether a classifier's justification matches human reasoning, not just whether its predicted label is correct.","The paper's stated next step of multi-label classification follows directly from its own perplexity guidelines, which instruct annotators to label all relevant dimensions and highlight the dominant one."],"supporting_citations":[{"why":"Defines the prior wellness-concept classification task on Reddit posts with four wellness dimensions, which this paper extends to all six dimensions and a different region.","marker":"[10]"},{"why":"Supplies the theoretical basis of Dunn's high-level wellness that grounds the annotation framework.","marker":"[11]"},{"why":"Provides a previous robustness and explainability study of wellness-dimension identification that motivates the evaluation design.","marker":"[12]"},{"why":"Identifies the Beyond Blue forums as the data source from which the 1,420 posts were collected.","marker":"[21]"},{"why":"Defines the Fleiss kappa inter-rater reliability measure used to report annotator agreement of 0.7592.","marker":"[22]"},{"why":"Introduces MentalBERT, the pretrained model that becomes the best-performing classifier in the comparison.","marker":"[27]"},{"why":"Supplies the LIME post-hoc explanation method used to evaluate interpretability against the annotated text spans.","marker":"[31]"}],"fun_headline_variants":["Holistix dataset: six wellness dimensions in mental-health posts","Dataset labels wellness dimensions in 1,420 mental-health posts","Mental-health posts classified into six wellness dimensions","Holistix: expert-labeled wellness spans in mental-health text","Holistix: 1,420 posts labeled for six wellness dimensions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark stands or falls on whether the expert-defined wellness taxonomy correctly captures the distress expressed in these posts, and on whether the two student annotators' labels, with agreement of 0.7592, are accurate enough to be treated as ground truth for training and evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Holistix dataset: six wellness dimensions in mental-health posts","Dataset labels wellness dimensions in 1,420 mental-health posts","Mental-health posts classified into six wellness dimensions","Holistix: expert-labeled wellness spans in mental-health text","Holistix: 1,420 posts labeled for six wellness dimensions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2972,"prompt_tokens":896,"completion_tokens":2076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1992}},"tokens_in":512,"tokens_out":2076,"duration_ms":14658,"temperature":1.0,"reasoning_tokens":1992,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:52:12.937030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 223 emotional posts and a sample of the 190 spiritual posts with a fresh group of clinical psychologists using the same published guidelines; if their labels agree with the released labels only at chance levels (for example, kappa below 0.4), or if the disagreement concentrates on the same ambiguous phrases flagged in the limitations, then the reported F1 scores for those dimensions measure label noise rather than model skill.","supporting_citations":[{"cited_title":"Wellxplain: Wellness concept extraction and classification in reddit posts for mental health analysis,","cited_arxiv_id":null,"evidence_quote":"Defines the prior wellness-concept classification task on Reddit posts with four wellness dimensions, which this paper extends to all six dimensions and a different region."},{"cited_title":"Wellness: Its origins, theories and current applications in the united states,","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical basis of Dunn's high-level wellness that grounds the annotation framework."},{"cited_title":"Welldunn: On the robustness and explainability of language models and large language models in identifying wellness dimensions,","cited_arxiv_id":null,"evidence_quote":"Provides a previous robustness and explainability study of wellness-dimension identification that motivates the evaluation design."},{"cited_title":"24/7 Support for Anxiety, Depression and Suicide Prevention. — beyondblue.org.au,","cited_arxiv_id":null,"evidence_quote":"Identifies the Beyond Blue forums as the data source from which the 1,420 posts were collected."},{"cited_title":"Mentalbert: Publicly available pretrained language models for mental healthcare,","cited_arxiv_id":null,"evidence_quote":"Introduces MentalBERT, the pretrained model that becomes the best-performing classifier in the comparison."},{"cited_title":"Explainable causal analysis of mental health on social media data,","cited_arxiv_id":null,"evidence_quote":"Supplies the LIME post-hoc explanation method used to evaluate interpretability against the annotated text spans."}],"review_version":1}