{"id":"4016d459-ebd4-4350-ab9c-c898a0c92d21","arxiv_id":"2502.06546","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A re-analysis of 66 survey responses finds that mood and demographics do not reliably predict users' software explanation needs.","lead":"This paper re-analyzes survey data from 66 software users to test whether mood, age, and gender predict how many or what type of software explanations people want. It finds no robust relationships: the two correlations it highlights do not survive the study's own multiple-comparison correction, so explanation needs appear highly personal.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under the paper's own Bonferroni rule (per-main-hypothesis, k=6), reactivity/UI p=.007 is below α=.0083 and H50 should be rejected, so the abstract's 'two significant correlations' and the text's 'none rejected' cannot both be right.","rationale":"The reader's verdict of REJECT is justified, but the load-bearing concern is not the sample representativeness highlighted in the reader's weakest_assumption. The decisive problem is internal: the paper's Bonferroni procedure, as stated in Section 3.4, contradicts both the abstract and the results section. If the per-main-hypothesis correction is applied with the actual number of subhypotheses in Table 5, the reactivity/UI correlation (p = .007) survives correction for H50, while the age/UI correlation (p = .044) does not survive correction for H30. The paper nevertheless says \"none of the main null hypotheses could be rejected\" and simultaneously advertises \"two significant correlations.\" There is no consistent statistical reading under which both statements are true: under per-family correction exactly one of the two correlations is significant; under a global 28-test correction neither is significant. Either way, the abstract's central claim is unsupported. This is a more direct threat to the paper's conclusion than the sampling concern, because it undermines the paper's own inferential logic rather than only its external validity. The rejection remains appropriate until the authors recompute with a pre-specified multiple-comparison procedure and align the abstract, results, and conclusions with the outcome.","tokens_in":10779,"tokens_out":7837,"duration_ms":68334,"concrete_test":"Reapply the Bonferroni correction exactly as specified in Section 3.4 to Table 5: count the subhypotheses under each main hypothesis H10–H50 (e.g., H50 has six: overall plus interaction, behavior, domain, security, UI), compute α_corr = 0.05/k for each family, and list which main hypotheses are rejected. If H50 is rejected and H30 is not, then the statements in Sections 4.3 and 5 (\"none of the main null hypotheses could be rejected\") and the abstract's \"two significant correlations\" are mutually inconsistent and must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is internally inconsistent about which correlations are significant. Section 3.4 (\"Correlation Analysis\") states that Bonferroni correction is applied per main hypothesis: if a main null hypothesis H_i0 is divided into subhypotheses, the corrected threshold is α_corr = 0.05/k, and rejection of any subhypothesis rejects H_i0. Under this rule, H50 (reactivity vs. explanation need) has six subhypotheses in Table 5 (overall need plus the five categories), so α_corr = 0.0083. The reported r = 0.33, p = 0.007 for reactivity and UI needs is below this threshold, so H50 should be rejected. Yet Section 4.3 and the Discussion state that \"none of the main null hypotheses could be rejected after applying the Bonferroni correction.\" Conversely, H30 (age vs. explanation need) also has six subhypotheses, so α_corr = 0.0083; the reported r = -0.25, p = 0.044 for age and UI needs is above this threshold and is not significant under the paper's own rule. Thus the abstract's assertion of \"two significant correlations\" cannot be derived from a single consistent application of the stated method: the reactivity correlation may survive a per-family correction, the age correlation does not, and if a global 28-test correction is intended then neither survives and the abstract's wording is still unsupported. The paper must specify which family is being controlled and recompute; as written, the reported significance decisions contradict the stated procedure.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a secondary analysis of an existing survey dataset from Droste et al. (RE'24) to investigate whether users' mood (sentiment and reactivity) and demographic factors (age, gender) correlate with their self-reported needs for explanations in software. The authors test 28 null hypotheses via Pearson correlations and Mann-Whitney-U tests, reporting two uncorrected significant correlations: reactivity with UI-related explanation needs (r=0.33, p=0.007) and age with UI-related explanation needs (r=-0.25, p=0.044). They conclude that explanation needs are highly subjective and only partially depend on objective factors, recommending that companies actively elicit user-specific explainability requirements. The paper's central statistical claims are, however, internally inconsistent: Section 4.3 states that no main null hypothesis survives Bonferroni correction, while the abstract and discussion present the two correlations as significant without qualification.","tokens_in":11089,"tokens_out":3713,"duration_ms":29178,"significance":"If the results were robust, the finding that mood and demographics are weak predictors of explanation needs would be a useful contribution to explainability requirements engineering, supporting calls for direct elicitation of user-specific explanation preferences. The paper's transparency—showing the full hypothesis table and correlation results, and discussing threats to validity—is a strength. However, the inconsistent application of multiple-testing correction undermines the specific empirical claims and the central message. The paper does not ship machine-checked proofs or reproducible code, but the statistical analyses are simple and re-computable from the reported numbers, which is a positive feature for verification.","major_comments":[{"comment":"The Bonferroni correction is applied inconsistently, making the reported significance decisions self-contradictory. The paper states in Section 3.4 that the correction is applied per main hypothesis, so for H30 (age vs. explanation need) and H50 (reactivity vs. explanation need), each of which has six subhypotheses, the corrected threshold is α_corr = 0.05/6 ≈ 0.0083. Under this rule, the reactivity-UI correlation (p=0.007) is below the threshold and should lead to rejection of H50, whereas the age-UI correlation (p=0.044) is above the threshold and would not reject H30. Therefore, Section 4.3's statement that \"none of the main null hypotheses could be rejected after applying the Bonferroni correction\" is wrong under the paper's own rule, and the abstract's \"Two significant correlations\" is not derivable from any single consistent application of the stated procedure. Please specify the correction family, recompute all decisions, and revise the abstract, Section 4.3, Section 5, and the conclusion accordingly.","section":"Section 3.4, Section 4.3, Table 5, Abstract"},{"comment":"The Mood Survey measures general, enduring mood traits (sentiment and reactivity) as acknowledged in Section 2.2, not momentary mood states. Parts of the discussion, such as \"situational emotional intensity may heighten users' demand for guidance\" and \"designing adaptive systems that respond dynamically to real-time emotional feedback, such as increased reactivity,\" treat reactivity as if it were a transient state. This overstates the applicability of the findings to mood-as-state and should be aligned with the trait-based nature of the instrument.","section":"Section 3.4, Section 5.2"},{"comment":"The paper excludes 17 of 83 participants due to incomplete mood data but provides no comparison of included versus excluded participants. The claim in Section 4.2 that \"we consider the sample is representative of the overall population in terms of expressed explanation needs\" is based solely on similarity to the original Droste et al. sample, which is tautological because the original sample is the parent dataset. This weakens the external validity claim and makes it difficult to assess whether the observed correlations and null results are biased by selective attrition. Please report reasons for missingness and compare demographic and explanation-need distributions between completers and non-completers.","section":"Section 3.3, Section 4.1, Section 5.4"}],"minor_comments":[{"comment":"The number of survey participants is reported as 83 in Section 3.3 but as 84 in Section 6; please reconcile this discrepancy.","section":"Section 3.3 vs. Section 6"},{"comment":"The scale for \"NFE by category X\" is listed as \"Ordinal 1..5,\" but the variable represents a count of explanation needs within a category; clarify whether this is a count or an ordinal rating.","section":"Table 2"},{"comment":"Please state whether the reported p-values are one-tailed or two-tailed, and specify this in the table caption or methodology section.","section":"Table 5"},{"comment":"The caption uses \"explainability needs\" while the text predominantly uses \"explanation needs\"; please use consistent terminology.","section":"Figure 1"},{"comment":"When describing the Bonferroni correction, the text gives only the example with two subhypotheses; it would be helpful to state explicitly that for hypotheses H30 and H50, which have six subhypotheses, the corrected threshold is 0.0083.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about arXiv:2502.06546. The re-analysis of Droste et al.'s survey data is legitimate, and the mostly null result is worth having: mood and demographics are weak predictors of explanation needs. But the paper contradicts its own significance testing, and that contradiction sits at the center of the headline findings.\n\nUnder the paper's stated Bonferroni rule (per main hypothesis), reactivity vs. UI explanations has p=.007 with six subhypotheses for H50, so alpha_corr=.0083 and H50 should be rejected. The age-UI correlation (p=.044) belongs to a larger family and does not survive any reasonable corrected threshold. So you can get at most one significant correlation from the paper's own method, not the two the abstract advertises. Meanwhile Section 4.3 says none of the main null hypotheses were rejected. Those three statements—two significant, one significant, none rejected—cannot all be true. That is load-bearing, because the paper's contribution is framed around those correlations.\n\nWhat the paper does well: it is a transparent secondary analysis of a published dataset, with clear provenance, a sensible threats section, and a practical conclusion that teams should gather explainability requirements directly rather than infer them from demographics or mood. The null result itself is useful to the explainability community. Credit is due for not overclaiming in the threats section, even though the abstract and discussion overclaim.\n\nSoft spots beyond the contradiction: Pearson on ordinal data is questionable at n=66; Spearman would be safer. The sample is 66 of 83, recruited from the authors' networks, with 17 dropped for incomplete mood data and no comparison of completers to non-completers. They acknowledge external validity, but it limits the strength of any positive finding. Also, \"explanation needs are very subjective\" overreaches a bit; null correlations with a few factors do not establish subjectivity, only that these factors aren't strong predictors.\n\nNet: this deserves a serious referee—the question is relevant and the negative result is informative. But the reporting of significance must be fixed before it can be published as is. The authors need to specify the family being controlled, recompute the corrected thresholds, and rewrite the abstract and results to match. I would not cite it in its current form.","headline":"Re-analysis of a published survey gives a mostly null result, but the paper's own Bonferroni rule makes its two headline correlations mutually inconsistent—a load-bearing error that a referee must catch.","tokens_in":11607,"tokens_out":6872,"would_cite":false,"duration_ms":60383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that users' need for software explanations is largely subjective: only emotional reactivity and age showed weak correlations, and only for user-interface explanations.","keywords":["explainability","explanation needs","mood analysis","sentiment","emotional reactivity","demographics","user interface explanations","correlation analysis"],"falsifier":"A preregistered replication on a larger, stratified sample (e.g., 500+ users) that measures mood immediately before a standardized software task, then records which explanation categories users request, would settle the claim. If the reactivity-UI correlation ($r=0.33$) and age-UI correlation ($r=-0.25$) do not reproduce, the paper's positive results are not stable; if mood or demographic factors predict other explanation categories, the conclusion that explanation needs are largely unpredictable would need revision.","tokens_in":10564,"feed_emoji":"🧠","tokens_out":7470,"duration_ms":60679,"temperature":0.7,"pith_summary":"This paper asks whether a software user's mood or demographic profile can tell you how many explanations they want and what kind. Analyzing survey data from 66 participants, it finds that the answer is mostly no: 26 of 28 tested relationships were not significant. The exceptions are both tied to user-interface explanations: people with more emotionally reactive moods reported more such needs ($r=0.33$, $p=0.007$), and older users reported fewer ($r=-0.25$, $p=0.044$). Stable mood (sentiment), gender, and all other explanation categories showed no reliable link. The authors conclude that explanation needs are highly subjective and depend only partially on objective factors, so software teams should elicit explainability requirements directly from users rather than infer them from profiles.","feed_headline":"Mood and age barely predict which explanations users want","feed_subtitle":"Only reactivity and age tie weakly to UI explanation needs; asking users beats guessing from profiles.","key_machinery":"The analysis is carried by two instruments joined by standard correlation tests. The first is the Mood Survey's two dimensions, sentiment (general positive or negative disposition) and reactivity (intensity and frequency of mood swings), measured on 7-point Likert scales. The second is the five-category taxonomy of explanation needs—interaction, system behavior, domain knowledge, privacy/security, and user interface—derived from the original survey. Relationships between continuous variables are tested with Pearson's $r$; gender differences are tested with the Mann-Whitney-U test; and multiplicity is controlled with Bonferroni correction.","core_discovery":"On its own terms, this paper's discovery is mostly an absence: the expected systematic relationships between mood, demographics, and explanation needs do not materialize. Across 28 null hypotheses, only two subhypotheses showed significant correlations, both confined to the user-interface category of the explanation-need taxonomy. Emotional reactivity correlated positively with the need for UI explanations, and age correlated negatively with that same category; gender, sentiment, and overall counts of explanation needs showed no significant associations. From this the paper concludes that explanation need is very subjective and cannot be reliably predicted from mood or demographic data, and that adaptive explanation systems should be driven by directly gathered user-specific requirements rather than inferred personas.","pith_inferences":["If the null pattern generalizes, the productive next variables to test are likely cognitive style, domain knowledge, and task context rather than demographics or mood, since those vary more within a person than between profiles.","The age-UI correlation may reflect generational interface expectations, but this sample has few participants in the middle age bands; a cohort-balanced replication would tell whether the effect is truly about age.","The two significant findings both concern UI explanations, suggesting that interface changes and confusing layouts are the place where personalization could pay off first."],"forward_implications":["Software companies cannot reliably build explainability features from age, gender, or mood profiles; user-specific elicitation remains necessary.","The only mood signal that mattered was reactivity, not stable sentiment, and only for UI explanations; adaptive systems would need real-time emotional state rather than static mood scores.","Age is a weak negative predictor of UI explanation needs, so younger users may need more interface guidance, but the signal is too small to drive policy alone.","Category-level analysis matters: total explanation-need counts hide the two correlations that appeared, so future studies should keep the taxonomy's categories separate."],"supporting_citations":[{"why":"Provides the survey dataset, the five-category taxonomy of explanation needs, and the category distribution that this re-analysis uses.","marker":"[17]"},{"why":"Supplementary material for the original survey containing the mood and reactivity items that are not in the published version; supplies the mood measurements re-analyzed here.","marker":"[18]"},{"why":"Source of the Mood Survey items for sentiment and reactivity used to classify participants' mood.","marker":"[38]"},{"why":"Defines the Pearson correlation coefficient used to test relationships between continuous variables.","marker":"[11]"},{"why":"Provides the Bonferroni correction applied to control Type I error across the multiple null hypotheses.","marker":"[21]"},{"why":"Supplies the goal-definition template and the classification of validity threats used in the study.","marker":"[42]"}],"fun_headline_variants":["Mood barely sways software explainability needs","Age, mood weakly tie to UI explanation needs","Software explanation needs resist mood and age prediction","Asking users beats guessing from mood or age"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 66 participants who completed the mood questionnaire represent software users generally; since the survey was spread through the authors' networks and a university message board, and 17 participants were dropped for incomplete mood data without a comparison to completers, a non-representative sample would change the pattern of correlations.","fun_headline_variants_meta":{"raw":{"variants":["Mood barely sways software explainability needs","Age, mood weakly tie to UI explanation needs","Software explanation needs resist mood and age prediction","Asking users beats guessing from mood or age"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1199,"prompt_tokens":895,"completion_tokens":304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":511,"tokens_out":304,"duration_ms":4835,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:06:10.204975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A preregistered replication on a larger, stratified sample (e.g., 500+ users) that measures mood immediately before a standardized software task, then records which explanation categories users request, would settle the claim. If the reactivity-UI correlation ($r=0.33$) and age-UI correlation ($r=-0.25$) do not reproduce, the paper's positive results are not stable; if mood or demographic factors predict other explanation categories, the conclusion that explanation needs are largely unpredictable would need revision.","supporting_citations":[{"cited_title":"In: RE’24","cited_arxiv_id":null,"evidence_quote":"Provides the survey dataset, the five-category taxonomy of explanation needs, and the category distribution that this re-analysis uses."},{"cited_title":"Ex- planations in Everyday Software Systems: Towards a Taxonomy for Explainability Needs","cited_arxiv_id":null,"evidence_quote":"Supplementary material for the original survey containing the mood and reactivity items that are not in the published version; supplies the mood measurements re-analyzed here."},{"cited_title":"Journal of Personality Assessment44(4) (1980)","cited_arxiv_id":null,"evidence_quote":"Source of the Mood Survey items for sentiment and reactivity used to classify participants' mood."},{"cited_title":"Noise reduction in speech processing (2009)","cited_arxiv_id":null,"evidence_quote":"Defines the Pearson correlation coefficient used to test relationships between continuous variables."},{"cited_title":"Springer (2012)","cited_arxiv_id":null,"evidence_quote":"Supplies the goal-definition template and the classification of validity threats used in the study."}],"review_version":1}