{"id":"388eef28-0231-44f7-a2c8-5bfef755d4b4","arxiv_id":"2606.19625","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In OLMo3-7B/Dolma3, social reasoning (SocialIQA) is supported by corpus regions distinct from STEM/knowledge benchmarks, with targeted unlearning validating the effect for SocialIQA but not for three of four benchmarks.","lead":"This paper traces which parts of a language model's training data support social reasoning versus STEM reasoning, using gradient-based attribution over a 576-cell topic-by-format taxonomy of OLMo3-7B's Dolma3 corpus. It finds social reasoning draws on distinct, more diffuse corpus regions, but causal validation via unlearning holds strongly for only one of the four benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reasoning-vs-knowledge contrast sharpness is confounded by query-count imbalance: SocialIQA has 10,000 queries vs ARC's 1,172, while MMLU pair is balanced; no query-matched control is reported.","rationale":"The reader's weakest-assumption focuses on the causal-validation link (unlearning damage on Base tracking Instruct-queried attribution), which is an important caveat but only weakens the validation pillar while leaving the descriptive provenance profile intact. The more load-bearing concern is a design confound in the 2x2 comparison: SocialIQA (10,000 queries) and ARC-Challenge (1,172) differ by ~8.5x in query count, whereas MMLU Social Sciences and MMLU STEM are almost exactly matched (~3,077 vs 3,018). Since influence is averaged over queries, the per-bin means for ARC have substantially larger query-induced variance, inflating the z-score spread and mechanically enlarging the SocialIQA–ARC apparent divergence. This directly affects the paper's headline RQ2 conclusion that the social–STEM contrast is sharper at the reasoning level than at the knowledge level. The bootstrap resampling over documents only (§3.3) cannot detect this query-effect because it conditions on the fixed query set. No query-matched control or query-subsample analysis is reported. Thus the concern is concrete, load-bearing, and distinct from the reader's chosen weak point. The verdict remains CONDITIONAL because the test could confirm robustness; but the paper as written overstates the reasoning-vs-knowledge contrast without addressing this confound.","tokens_in":62004,"tokens_out":9477,"duration_ms":106108,"concrete_test":"Subsample SocialIQA's 10,000 queries to 1,172 (matching ARC) using the same OLMES protocol, recompute the bin-level influence aggregation, and re-measure the reasoning-pair contrast (e.g., max topic |Δ| and full-grid profile divergence) against the knowledge pair. Repeat across at least 20 random query subsets to obtain a distribution of the sharpness metric. If the reasoning-vs-knowledge gap shrinks to the MMLU-pair level, the RQ2 claim is an artifact of query-count imbalance; if it persists, the finding is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RQ2/abstract claim that the social–STEM separation is sharper for reasoning than for knowledge rests on comparing the SocialIQA–ARC-Challenge pair with the MMLU Social Sciences–MMLU STEM pair. Influence per document is averaged over benchmark queries (Eq. in §3.3). ARC-Challenge has 1,172 queries, SocialIQA has 10,000, while MMLU-SS/STEM have ~3,077/3,018, nearly balanced. With many fewer queries, ARC's per-document influence estimates carry much larger query-induced variance, making its 576-bin means noisier and its within-benchmark z-scores overdispersed. This mechanically inflates the SocialIQA–ARC profile difference relative to the MMLU pair. Table 11 quantifies the contrast via max topic |Δ| (0.91 reasoning vs 0.63 knowledge), a metric directly sensitive to z-score spread. The paper's bootstrap (§3.3) resamples documents within bins while holding the query set fixed, so it cannot detect or correct for this variance inflation. No query-subsample analysis or matched-query robustness check appears in the paper or appendices. This threatens the central descriptive conclusion itself, not just the causal-validation pillar.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a capability-provenance method: gradient-based training-data attribution (TrackStar via Bergson) on OLMo3-7B/Dolma3, aggregated over WebOrganizer's 24x24 topic-format taxonomy (576 bins), and applied in a 2x2 design crossing domain (social vs. STEM) with capability type (reasoning vs. knowledge): SocialIQA, MMLU Social Sciences, ARC-Challenge, MMLU STEM. The central descriptive finding is that SocialIQA's 576-bin influence profile is a structural outlier, correlating at r <= 0.22 with the three comparison benchmarks versus 0.53-0.86 among them (permutation p ~ 1e-4), with Literature x Customer Support extreme-positive for SocialIQA (+16.0 z) and negative for both MMLU benchmarks, and with the social-STEM separation reported as sharper at the reasoning level (max topic |Delta| = 0.91) than at the knowledge level (0.63). NGDiff targeted unlearning of high-attribution documents within topics degrades SocialIQA more than random in-topic controls (paired Wilcoxon BH-adjusted p ~ 1e-5), with weaker or null results for the other benchmarks. Lexical profiling, correctness differentials, and held-out probes (BBQ, ToMBench, BBH, etc.) provide corroborating breadth. Code, sampling manifests, the bin-level influence matrix, and unlearning checkpoints are released.","tokens_in":2258,"tokens_out":2427,"duration_ms":226323,"significance":"If the descriptive findings survive robustness checks, this is a useful, well-scoped contribution: it converts noisy document-level attribution into stable, named corpus regions and produces specific, falsifiable findings (the Literature x Customer Support bin as a social-reasoning-supporting region; its correctness-differential sign flip between reasoning and knowledge benchmarks). The paper earns explicit credit for an open, reproducible pipeline (detailed compute accounting, sampling manifests, released influence matrix and checkpoints), a genuine 2x2 factorial design, and notably candid reporting: Section 4.3 and Section 5 state that unlearning validation is null for MMLU Social Sciences and negative for ARC-Challenge, and G.8 openly reports that attribution and unlearning damage do not co-rank for held-out ToM probes. The descriptive claim is supported by several independent aggregations (full-grid correlations, marginals, permutation test). The significance is tempered by two gaps: the RQ2 'sharper' claim is potentially confounded by benchmark query-count imbalance, and the causal pillar currently supports only SocialIQA among the four primary benchmarks. The paper's value is","major_comments":[{"comment":"RQ2's claim that the social-STEM split is sharper at the reasoning level compares SocialIQA-ARC-Challenge with MMLU SS-MMLU STEM. Per-bin means average influence over benchmark queries (Section 3.3); ARC-Challenge has 1,172 queries vs SocialIQA's 10,000, while the MMLU pair is nearly balanced (3,077/3,018) (Table 8). ARC's bin means therefore carry roughly sqrt(10000/1172) ~ 2.9x more query-sampling variance, which survives within-benchmark z-standardization and inflates |Delta z| for the reasoning pair. Table 11's max-topic metric (0.91 vs 0.63) is exactly the kind of max statistic that variance overdispersion inflates. Both bootstrap schemes (Section 3.3 document-resampling; D.1 bin-resampling) hold the query set fixed, so this variance is unbounded in the paper. Figure 15's pattern (ARC correlating 0.53/0.55 with the MMLUs vs 0.86 between the MMLUs) is consistent with ARC being noise-","section":"Section 3.3 (aggregation rule), Table 8, Table 11, Appendix D.1"},{"comment":"The causal-validation pillar assumes NGDiff unlearning damage on OLMo3-7B Base tracks the Instruct-queried attribution signal. For the primary benchmarks, validation is significant only for SocialIQA; MMLU Social Sciences is null, ARC-Challenge has a negative median paired difference, and MMLU STEM is weak (Section 4.3). For held-out ToM probes, G.8 reports that gradient attribution and unlearning damage do not co-rank (topic-level Spearman ~ 0) - a direct failure of that premise on the only held-out causal test. The paper is candid about this (G.8, Section 5), but Section 4.3's RQ3 takeaway ('usually causes larger accuracy drops') and the abstract's 'partial causal validation' somewhat overstate coverage: as reported, the causal pillar supports SocialIQA specifically, with the premise contradicted elsewhere. Recommend: (i) bring the G.8 non-convergence into the main-text RQ3 discussion;","section":"Section 4.3, Appendix G.8, Section 5"},{"comment":"The abstract states: 'We validate on other open-data model, Comma v0.1 7B-2T (Common Pile) and DCLM-Baseline-7B (DataComp-LM): causal selectivity holds on both models, while the provenance map is ecosystem-specific.' I could not find these experiments anywhere in the full text, the appendices, the resource accounting, or the released-artifact description; the full-text abstract itself omits this sentence. A claim of cross-ecosystem causal selectivity is load-bearing for the abstract's credibility and is not verifiable from the manuscript. Either add the experiments (with the same detail as Appendix J) or remove/correct the claim.","section":"Abstract (arXiv metadata); full text; appendices"}],"minor_comments":[{"comment":"Section 4.1 says SocialIQA's profile correlates with the three comparison benchmarks 'at r <= 0.22 versus r = 0.76-0.86 among them,' but Appendix D.1 and Figure 15 report the among-comparison correlations as 0.53, 0.55, 0.86. Align the main text with the appendix range.","section":"Section 4.1 vs Appendix D.1 / Figure 15"},{"comment":"Section 3.4.1 specifies forget-set size k = 2,000, while Section 4.3's headline result is described as 'the faithful top-200 per-document replication.' State which k underlies Table 49/Figure 42 and where the k = 2,000 runs are reported.","section":"Section 3.4.1 vs Section 4.3"},{"comment":"Figure 12's axis labels abbreviate topic names ('Sci. & Tech.', 'Social', 'Software Dev.') inconsistently with Table 2's canonical names (Science & Technology, Social Life, Software Development); add a caption note or use the canonical names.","section":"Figure 12 / Appendix A"},{"comment":"The claim of being 'the first application of gradient-based corpus-scale TDA to capability provenance with taxonomic aggregation' is broad; soften or delimit it against the cited prior work (e.g., Ruis et al. 2025, Zhao et al. 2024).","section":"Section 2"},{"comment":"Literature x Customer Support has a reported z = +16.0 for SocialIQA. Because this bin is in the corpus's token tail (Table 6) and the working set draws 10,000 of its ~34K documents, show the within-bin influence distribution (or a document-downsampling check) to rule out a small set of outlier documents driving the extreme mean.","section":"Figure 2 / Table 6 / Appendix H"},{"comment":"Section 4.3 reports p ~ 1e-5 (Wilcoxon BH-adjusted) while Section 5 reports pBH < 1e-4 for the SocialIQA validation; label the test, conditioning, and multiplicity correction identically so the reader can compare.","section":"Section 4.3 vs Section 5"}],"recommendation":"major_revision","confidential_remarks":"The recommendation is driven by three fixable issues: (1) the RQ2 'sharper at reasoning' claim is confounded by a 8.5x query-count imbalance between SocialIQA and ARC-Challenge relative to the balanced MMLU pair, and no query-matched control appears anywhere; (2) the arXiv-metadata abstract claims cross-model validation on Comma v0.1 7B-2T and DCLM-Baseline-7B that is entirely absent from the full text; (3) the causal-validation premise is contradicted by Appendix G.8's Spearman ~ 0 result for held-out ToM probes and supports only SocialIQA among the primary benchmarks. The team deserves credit for transparent null reporting and a genuinely open pipeline; I hope the revision retains that candor while adding the requested robustness analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper delivers the first bin-level capability-provenance map on a fully open model+corpus, with all artifacts released—that's a real, reusable instrument. But the headline contrast—sharper at the reasoning level than the knowledge level—rests on comparing SocialIQA (10,000 queries) with ARC-Challenge (1,172 queries), and that imbalance inflates the difference. The causal validation is partial at best: it works for SocialIQA, is null for MMLU Social Sciences, negative for ARC-Challenge, and the authors themselves report that attribution and unlearning don't co-rank on held-out ToM probes.\n\nThe novel part is the pipeline. Aggregating TrackStar/Bergson influence into WebOrganizer bins, with stratified sampling, a 2x2 factorial design, and a released 576x4 influence matrix, is a step forward. The correctness differential (Literature x Customer Support flipping sign between reasoning and knowledge benchmarks) is a nice diagnostic. The authors are unusually transparent: they state the nulls, the Base/Instruct checkpoint mismatch, and the identity second-order approximation. That honesty earns credit.\n\nThe soft spot is real. With ARC at 1,172 queries versus SocialIQA's 10,000, ARC's per-document influence estimates carry much larger query-sampling noise. The paper's bootstrap resamples documents within bins but holds the query set fixed, so it cannot see this variance. The result is that ARC's bin-level z-scores are inflated relative to signal, mechanically widening the SocialIQA-ARC gap compared to the well-matched MMLU pair. The stress-test note is right to call this a threat to the central descriptive conclusion, not just the causal pillar. I think the core outlier claim for SocialIQA may survive—it also differs from the two 3k-query MMLU benchmarks—but the 'sharper for reasoning than knowledge' conclusion is not supported as reported. A simple query-subsample analysis (downsample SocialIQA to ARC's count, or bootstrap over queries) would settle it.\n\nThe paper deserves serious peer review. The method is new, the artifacts are reproducible, and the descriptive map is useful for data auditing even if the causal claims need narrowing. A referee should ask for query-matched robustness checks and a revised abstract that limits the validation claim to SocialIQA unless the other benchmarks come up.\n\nWho benefits: the interpretability and data-auditing community. I'd bring it to a reading group and would cite the method.","headline":"A genuine new instrument for capability provenance—but the sharper-for-reasoning claim is confounded by query-count imbalance, and the validation holds for SocialIQA only.","tokens_in":62860,"tokens_out":4967,"would_cite":true,"duration_ms":49813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Social and STEM reasoning in a language model draw support from different, nameable regions of the pretraining corpus — and the split is sharper for reasoning than for factual knowledge.","keywords":["training-data attribution","capability provenance","social reasoning","machine unlearning","pretraining corpus","topic-format taxonomy","benchmark contrast","open-data language models"],"falsifier":"Pretrain or continue-pretrain a same-scale model with the top-attributed SocialIQA documents (concentrated in Literature × Customer Support) held out of the mix, alongside a matched random holdout; if the benchmark drop does not reproduce, the attribution map is not causal. A cheaper check: restrict the SocialIQA forget set to top-attributed documents drawn outside the Literature topic — if the Wilcoxon effect vanishes, the result is topic membership rather than attribution ranking.","tokens_in":61758,"feed_emoji":"🧭","tokens_out":9590,"duration_ms":92307,"temperature":0.7,"pith_summary":"This paper tries to establish where a language model's capabilities come from in the pretraining corpus — specifically, whether social reasoning and STEM reasoning are fed by different, nameable regions of the training data. It maps gradient-based attribution scores onto a 576-cell topic-by-format taxonomy of the Dolma3 corpus and finds SocialIQA is the structural outlier: its support profile barely correlates with the three comparison benchmarks (r ≤ 0.22 vs 0.53–0.86 among them), and its strongest positive region, Literature × Customer Support, is strongly negative for the knowledge and STEM benchmarks. The social–STEM split is sharper for reasoning than for knowledge, and a single corpus bin flips sign between reasoning and knowledge in a correctness analysis. Targeted machine unlearning gives partial causal backing: forgetting high-attribution bins degrades the aligned benchmark more than random in-topic forgetting, clearly for SocialIQA (BH-adjusted p ≈ 1e-5), and this selectivity — though not the exact map — replicates on two other open-data models. If right, the paper supplies an inspectable map of which kind of text feeds which capability, with direct uses in data curation and capability auditing.","feed_headline":"Social reasoning traces to dialogue-heavy pretraining data","feed_subtitle":"A 576-bin corpus map finds narrative text feeds social reasoning; targeted forgetting reproduces the effect.","key_machinery":"The load-bearing unit is the WebOrganizer bin: one topic × format cell in a 24-topic × 24-format taxonomy (576 named regions, e.g. Literature × Customer Support). Noisy document-level gradient attribution (TrackStar influence on base-model gradients, queried through the instruction-tuned checkpoint) is aggregated into these bins, and each benchmark's profile is z-scored across all 576 cells so profiles can be compared by position rather than magnitude. The 2×2 benchmark design — domain (social vs STEM) crossed with capability type (reasoning vs knowledge) — supplies the contrastive frame, and NGDiff LoRA unlearning on the base model supplies the interventionist check of whether high-attribut","core_discovery":"The central claim: in OLMo3-7B, social and STEM reasoning draw on qualitatively different corpus regions, and the contrast is stronger for the reasoning pair (SocialIQA vs ARC-Challenge) than for the knowledge pair (MMLU Social Sciences vs MMLU STEM). SocialIQA's 576-bin influence profile correlates with the three comparison benchmarks at r ≤ 0.22, while those three correlate at r = 0.53–0.86 among themselves. The most discriminative bin, Literature × Customer Support, is extreme-positive for SocialIQA (+16.0 z) and strongly negative for MMLU Social Sciences (−7.31) and MMLU STEM (−5.75); it is also the top correctness-differential cell for both reasoning benchmarks and the strongest error c","pith_inferences":["The Literature × Customer Support result suggests a genre-level hypothesis the paper leaves implicit: Q&A-style talk about characters and stories may train the model to track minds, not to store facts. A testable extension is a small-scale pretraining run that up-weights character-dialogue Q&A and checks held-out Theory-of-Mind gains.","The paper's G.8 finding — topical unlearning works for Theory-of-Mind while gradient attribution and unlearning damage do not co-rank — hints at a gap between gradient-aligned (typical) support and causally necessary (critical) data; separating the two could be its own measurement task.","The ARC-Challenge and MMLU Social Sciences validation failures suggest the causal leg is strongest where influence concentrates in a few unusual bins; for benchmarks with diffuse support, document-level ranking within a broad topic may be the wrong granularity, and a coarser or hierarchical bin structure could be tested.","If provenance maps are ecosystem-specific while causal selectivity transfers, social reasoning looks less like stored social knowledge and more like a reading style — implying curators could steer it with generic narrative data rather than domain-specific documents."],"forward_implications":["Social reasoning in OLMo3-7B is carried heavily by short, dialogue-rich narrative text — the Literature × Customer Support bin is its single strongest support region and is simultaneously a negative influence on knowledge and STEM benchmarks.","Corpus support separates by capability type, not only domain: the social–STEM split is sharper between SocialIQA and ARC-Challenge than between their knowledge counterparts.","The same corpus content can help correct reasoning and hurt correct factual recall: Literature × Customer Support is the top correctness bin for both reasoning benchmarks and the strongest error bin for both knowledge benchmarks.","Forgetting high-attribution bins produces a selective damage profile rather than generic degradation, and this selectivity — though not the specific bin map — replicates across three open-data model families.","High-influence bins carry a consistent lexical signature — short interpersonal text alongside long-form documentation — so capability-relevant corpus regions are recognizable from cheap surface text features."],"fun_headline_variants":["Social reasoning traces to narrative-heavy pretraining data","Pretraining map: social reasoning leans on literature","576-bin map spotlights distinct social vs STEM data","Forgetting lit data weakens social reasoning, not STEM","Social reasoning's data fingerprint: literature and support"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The causal-validation leg assumes that how much unlearning damage a document causes tracks how strongly that document's gradient aligns with the benchmark's queries; the paper's own results show this holds clearly for SocialIQA but is null for MMLU Social Sciences, has a negative paired difference for ARC-Challenge, and does not co-rank for held-out Theory-of-Mind probes — if unlearning damage does not index the attribution signal, only the descriptive provenance profile rema","fun_headline_variants_meta":{"raw":{"variants":["Social reasoning traces to narrative-heavy pretraining data","Pretraining map: social reasoning leans on literature","576-bin map spotlights distinct social vs STEM data","Forgetting lit data weakens social reasoning, not STEM","Social reasoning's data fingerprint: literature and support"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2068,"prompt_tokens":852,"completion_tokens":1216,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1142}},"tokens_in":596,"tokens_out":1216,"duration_ms":10478,"temperature":1.0,"reasoning_tokens":1142,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:40:00.472042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pretrain or continue-pretrain a same-scale model with the top-attributed SocialIQA documents (concentrated in Literature × Customer Support) held out of the mix, alongside a matched random holdout; if the benchmark drop does not reproduce, the attribution map is not causal. A cheaper check: restrict the SocialIQA forget set to top-attributed documents drawn outside the Literature topic — if the Wilcoxon effect vanishes, the result is topic membership rather than attribution ranking.","supporting_citations":[],"review_version":2}