{"id":"d9aa1cf6-abe5-49d7-ae22-b2601c01958e","arxiv_id":"2608.10974","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MUSE curates 579 expert-annotated paragraphs and a pipeline that extracts 36,960 problem-solution-rationale triplets from full-text arXiv papers, with evidence that rationale supervision helps LLMs on complex problems but harms them on trivial ones.","lead":"This paper introduces MUSE, a dataset and extraction pipeline that turns scientific papers into structured problem-solution-rationale triplets, and uses it to build a knowledge base of about 37,000 entries across arXiv fields. It also reports a preliminary result that using rationales to train an LLM helps on complex multi-constraint problems but hurts on simpler ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rationale-supervision finding in §6.2 is confounded: Table 8 compares PRS against the untuned base model, not against the solution-only PS condition, so the cross-over effect may reflect fine-tuning rather than rationales.","rationale":"The central claim of the paper, highlighted in the abstract and introduction, is that rationale supervision has a conditional effect: helpful for complex, multi-constraint problems and harmful for simple ones. The experiment designed to support this claim compares the rationale-supervised model (PRS) to the untuned base model in the stratified analysis (Table 8), rather than to the solution-only model (PS). This is a control-condition error: PS and PRS differ only in whether the rationale is included, so any difference between them isolates the effect of rationales. Comparing PRS to Base conflates rationale supervision with the overall effect of fine-tuning. Since Table 12 shows Base beats both PS and PRS on overall quality, the observed cross-over in the strata may simply reflect that fine-tuning helps complex problems and hurts trivial ones, independent of rationales. The paper's qualitative explanation in Appendix D attributes the effect to rationale-induced constraint reasoning, but without the PS control this is speculative. This is addressable by re-running the stratified evaluation, so the appropriate verdict remains conditional.","tokens_in":14093,"tokens_out":5601,"duration_ms":49780,"concrete_test":"Re-run the stratified evaluation with all three conditions (PS, PRS, Base) on the same 10% complex and 10% trivial strata using the identical LLM judge and complexity ratings from Claude Opus 4.8. If PS shows the same cross-over pattern (better than Base on complex, worse on trivial), the effect is a fine-tuning artifact and the rationale-specific claim fails. If the cross-over is unique to PRS, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 reports that on the 10% most complex problems the rationale-supervised model (PRS) beats the base model, and on the 10% most trivial problems it loses (Table 8). But the table's columns are 'Rationale' and 'Base', not 'PRS' and 'PS': the solution-only model that isolates the effect of adding rationales is absent from the stratified analysis. The paper's full results (Table 12) show the untuned base model outperforms both PS and PRS on overall quality, so the stratified cross-over could be driven by SFT+GRPO training generally, not by the rationale field. Without stratified PS numbers, the headline claim that 'rationale supervision is useful when a problem contains multiple constraints... but can be harmful on simple problems' is not identified. The low specificity (2.83) and completeness (3.12) of the generated rationales in Table 7 further compound this, since the training signal itself is of uncertain quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MUSE, a full-text, cross-domain resource of scientific Problem–Solution–Rationale (P–S–R) triplets. It defines an annotation schema covering paragraph relevance, salient problem/solution/rationale spans, solves and rationale_of relations, and conceptual coreference; curates 579 expert-annotated paragraphs; and builds a modular extraction pipeline consisting of cascade paragraph filtering, token-level span classification, Mistral-7B relation extraction, GPT-4o post-processing, and Claude Opus refinement. The pipeline is used to produce a knowledge base of 36,960 triplets from arXiv full text. The paper also reports a preliminary study in which two LLMs are trained with problem-to-solution (PS) or problem-to-rationale-to-solution (PRS) supervision, and claims that rationale supervision helps on complex, multi-constraint problems but hurts on simple ones.","tokens_in":14308,"tokens_out":3804,"duration_ms":39778,"significance":"If the resource quality claims hold, MUSE would be a valuable addition to scientific IE: it targets a structure—local problem–solution pairs with author-stated rationales—that abstracts-level resources do not capture, and it provides source provenance and multiple annotation layers. The seed annotation has high human agreement, the pipeline is decomposed into components that are each evaluated against gold spans and relations, and the authors commit to releasing the dataset, models, and KB. The conditional finding about rationale supervision is potentially interesting for LLM post-training. However, the final rationales are LLM-generated and receive only moderate specificity and completeness scores, and the stratified downstream comparison that underlies the headline finding omits the solution-only control, so both the KB-quality and rationale-supervision claims need additional support before they should be taken as established.","major_comments":[{"comment":"The paper's headline claim—that rationale supervision is useful on complex problems but harmful on simple ones—is not identified by the reported stratified comparison. Table 8 compares columns labeled 'Rationale' and 'Base', not 'PRS' and 'PS'. The appropriate control for isolating the effect of adding rationales is the solution-only PS variant, which is absent from the top/bottom 10% strata. Since Table 12 shows that the untuned base model has the highest absolute overall scores for Qwen-3-32B (7.12 vs. 6.41 for PRS and 5.12 for PS), the crossover in Table 8 could be driven by SFT+GRPO training in general rather than by the rationale field. Please report stratified PS versus PRS results for the complex and trivial strata, ideally with confidence intervals or significance tests, and revise the interpretation accordingly if the crossover does not persist against PS.","section":"§6.2, Table 8"},{"comment":"The KB-level claim of high quality rests on thin validation of the final rationale fields. Rationale specificity is 2.83 and completeness is 3.12 on a 1–5 scale, and the human validation of the GPT-4o rationale construction step covers only 30 examples. Because every one of the 36,960 triplets contains a generated and refined rationale, a 30-example evaluation is not sufficient to estimate the precision of the final KB. Please provide a larger, preferably stratified human evaluation of the final triplet fields (problem, solution, rationale) sampled from the constructed KB, with an error taxonomy; this is load-bearing for the resource contribution.","section":"§5.3, §5.4, Table 7"},{"comment":"The Claude Opus 4.8 refinement step is introduced to improve separation between solution and rationale fields, but its evaluation is a manual review of 50 records reporting an 86% 'clear separation' rate. The criterion for clear separation is not defined, and no agreement measure is reported for this judgment. Since this refinement is applied to the entire KB, please specify the rating protocol and, ideally, have a second annotator judge a subset to establish reliability.","section":"§5.3"}],"minor_comments":[{"comment":"The abstract states '37K P–S–R triplets' while Section 5.4 reports 36,960 triplets; please make these numbers consistent.","section":"Abstract / §5.4"},{"comment":"The paper reports that human agreement is 'very high' and gives per-class P/R/F1 in the human-agreement rows, but it would be useful to state the number of paragraphs in the quality-control double-annotation subset and to report an overall agreement statistic such as Cohen's kappa for paragraph relevance and span-level annotation.","section":"§3.2"},{"comment":"Claude Opus 4.8 is used both to refine the KB's rationale fields (Section 5.3) and as the judge and complexity scorer in the downstream study. The paper should state explicitly whether this overlap could bias the evaluation, since the refinement may imprint stylistic regularities that the same model family could reward.","section":"§6.1"},{"comment":"The multi-class Cs-RoBERTa rationale row appears to contain an unusually formatted entry ('0.29 0.00 0.01'); please check the table for typesetting errors, as several rationale rows contain near-zero F1 values that are worth discussing explicitly in the text.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The resource is potentially solid and the seed annotation is a real contribution, but the current downstream analysis does not support the headline rationale-supervision claim because the stratified comparison lacks the solution-only control. The evaluation of generated rationales is also too small relative to the scale of the KB. I would encourage the editor to request the stratified PS/PRS results and a larger KB-level validation sample; with those additions, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MUSE knowledge base is a genuine contribution and deserves a serious referee. What's actually new is a full-text, cross-domain dataset where problem, solution, and rationale spans are linked by solves and rationale_of edges, with conceptual coreference; no prior resource does this at paragraph level. The 579-paragraph seed has high human agreement, the modular pipeline is sensible (filtering, span classification, relation extraction, post-processing), and each component is evaluated against gold annotations. The failure-mode analysis for end-to-end LLM extraction is useful and explains the design.\n\nThe soft spots are real but mostly where the paper already says it is preliminary. The generated rationales are the weakest field: specificity 2.83 and completeness 3.12 on a 1–5 scale, judged on only 30 examples. Since rationales are the distinguishing part of the KB, this matters, but the authors are transparent about it and the seed annotations provide an upper bound.\n\nThe bigger issue is the rationale-supervision finding. Section 6.2 says that on complex problems PRS beats base and on trivial problems it loses (Table 8), and interprets this as rationale supervision being conditional on problem structure. But Table 8's columns are Rationale and Base, not PRS and PS. The untuned base outperforms both trained variants in absolute terms in Table 12, so the cross-over could reflect fine-tuning generally rather than the rationale field. The paper does show PRS > PS overall for Qwen (6.41 vs 5.12), so rationales have some effect, but the complexity interaction is not identified without stratified PS numbers. The paper's own 'preliminary' caveat in the Limitations partially covers this, but the abstract states the finding fairly flatly.\n\nMinor: LLM-judge scores lack confidence intervals or significance tests, and Claude Opus is used both to refine records and to judge the downstream study, a mild leakage. Also, KB-level precision of the 36,960 triplets is not directly measured beyond the component evaluations.\n\nWho this is for: anyone building scientific IE resources or using them for retrieval/training. The KB and seed are the value; the application study should be treated as illustrative.\n\nI would send this to peer review, asking the authors to stratify the analysis by training condition (PS vs PRS vs base) and to report uncertainty on the judge scores. The resource is solid enough that these are fixable rather than fatal.","headline":"MUSE's KB is a genuine resource worth peer review; the rationale-supervision finding is under-identified and needs a fix before it can be treated as a result.","tokens_in":14828,"tokens_out":2524,"would_cite":true,"duration_ms":24522,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-grained scientific reasoning can be mined from full text as Problem–Solution–Rationale triplets at scale, and that using these rationales to train LLMs helps on complex, multi-constraint problems but hurts on…","keywords":["scientific information extraction","Problem-Solution-Rationale","full-text mining","knowledge base construction","rationale supervision","LLM fine-tuning","cross-domain corpus","arXiv"],"falsifier":"Take a random sample of 300 triplets from the 36,960, have independent expert annotators read the source paragraphs, and count how many rationale fields faithfully represent the authors' stated reasoning; if the faithful proportion falls well below the paper's reported groundedness scores (for instance, below 60%), the central claim weakens. Separately, retrain the PRS model with verbatim rationale spans instead of generated ones and check whether the complex-problem advantage persists.","tokens_in":13906,"feed_emoji":"🧠","tokens_out":4353,"duration_ms":36982,"temperature":0.7,"pith_summary":"This paper introduces MUSE, a knowledge base of 36,960 Problem–Solution–Rationale (P–S–R) triplets mined from full-text arXiv papers across all scientific domains. The authors curate 579 expert-annotated paragraphs, design a modular extraction pipeline that first filters paragraphs, then extracts and links problem and solution spans, and finally generates self-contained problem, solution, and rationale fields. They evaluate each component and use the KB to test whether training an LLM to output a rationale before a solution improves scientific problem-solving. They find that rationale supervision helps on complex, multi-constraint problems but hurts on trivial ones. If correct, MUSE offers a new cross-domain resource for scientific IE, and the conditional finding suggests explanation-style supervision should be deployed selectively.","feed_headline":"Rationale training helps LLMs only on complex problems","feed_subtitle":"A new 37K-triplet knowledge base of scientific problems, solutions, and rationales drives the finding.","key_machinery":"The central object is the Problem–Solution–Rationale (P–S–R) triplet, where a problem is a limitation, obstacle, gap, or desideratum; a solution is a method, mechanism, design choice, or procedure used to address it; and a rationale is the author-stated motivation linking the two. The carrying mechanism is the modular extraction pipeline: a cascade of paragraph classifiers, token-level span extraction with DeBERTa-v3-large, relation and coreference classifiers based on Mistral-7B, and an LLM post-processing step (GPT-4o, refined by Claude Opus) that turns span annotations into standalone fields. The downstream experiment uses supervised fine-tuning followed by Group Relative Policy Optimization with a composite reward (correctness, format, divergence) to compare solution-only (PS) versus rationale-then-solution (PRS) training.","core_discovery":"The paper claims that local technical reasoning in scientific papers can be captured as structured triplets: a concrete problem (a limitation, obstacle, gap, or desideratum), the solution used to address it (a method, mechanism, design choice, or procedure), and the author-stated rationale connecting the two. It further claims that this structure can be recovered at scale with a modular pipeline that outperforms end-to-end LLM extraction, and that the resulting rationales provide a useful training signal for LLM-based scientific problem solving. The load-bearing empirical finding is that rationale supervision improves solution generation on problems with multiple interacting constraints, because it keeps all constraints in view, but degrades performance on simple problems, because the model over-engineers responses to right-sized tasks.","pith_inferences":["The conditional finding suggests an adaptive training curriculum that adds rationale supervision only when a problem's constraint load is high; this could be tested by routing training examples by the paper's complexity score.","The KB could support cross-domain analogical ideation by retrieving problems with similar rationales but different domains, a use the paper lists but does not test.","The cascade design implies that end-to-end LLM extraction fails not on raw span quality but on relational alignment, a lesson likely to transfer to other multi-entity extraction tasks beyond science."],"forward_implications":["If the central claim holds, MUSE can serve as training and evaluation data for scientific information extraction and LLM-based problem solving across all arXiv domains.","Rationale supervision becomes a tunable component of LLM post-training: adopt it for constraint-dense problems, omit it for routine ones.","The pipeline's decomposition (filter, span extract, link, rewrite) provides a recipe for scaling expert annotation to large full-text corpora beyond this task.","The KB enables meta-scientific analyses of how different fields formulate constraints and justify methodological choices.","The conditional result gives a concrete hypothesis for future reasoning-model training: explanation-style supervision is beneficial only when the problem's structure demands it."],"supporting_citations":[{"why":"Supplies the unarXive source corpus from which full-text paragraphs are sampled for annotation and large-scale KB construction.","marker":"Saier and Färber, 2023"},{"why":"DeBERTa-v3-large is the selected encoder for problem and solution span extraction.","marker":"He et al., 2021"},{"why":"Mistral-7B-Instruct is fine-tuned for conceptual coreference and solves-relation extraction.","marker":"Jiang et al., 2023"},{"why":"SciBERT provides the embeddings used in BERTScore for span matching and rationale evaluation.","marker":"Beltagy et al., 2019"},{"why":"GRPO is the reinforcement-learning algorithm used in the downstream rationale-supervised fine-tuning study.","marker":"Shao et al., 2024"},{"why":"BERTScore is the metric used to compare extracted rationales against human rationales.","marker":"Zhang et al., 2020"},{"why":"TeamTat is the annotation platform used for expert annotation of the seed dataset.","marker":"Islamaj et al., 2020"}],"fun_headline_variants":["Rationale training helps LLMs on complex problems, hurts on easy ones","Rationale supervision: helps multi-constraint problems, hurts simple ones","LLMs trained on scientific rationales: better on complex, worse on easy","37K triplet KB reveals rationale training's split effect on LLMs","New 37K scientific solution KB shows why rationale training cuts both ways"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generated rationale fields are not quoted from the papers but rewritten by a language model from extracted anchors, and they are checked by humans on only 30 examples; if those fields misrepresent what the authors actually said, both the knowledge base and the training signal built on it lose their grounding.","fun_headline_variants_meta":{"raw":{"variants":["Rationale training helps LLMs on complex problems, hurts on easy ones","Rationale supervision: helps multi-constraint problems, hurts simple ones","LLMs trained on scientific rationales: better on complex, worse on easy","37K triplet KB reveals rationale training's split effect on LLMs","New 37K scientific solution KB shows why rationale training cuts both ways"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001093,"raw_usage":{"total_tokens":4516,"prompt_tokens":851,"completion_tokens":3665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":3583}},"tokens_in":467,"tokens_out":3665,"duration_ms":26381,"temperature":1.0,"reasoning_tokens":3583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:02:22.245802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 300 triplets from the 36,960, have independent expert annotators read the source paragraphs, and count how many rationale fields faithfully represent the authors' stated reasoning; if the faithful proportion falls well below the paper's reported groundedness scores (for instance, below 60%), the central claim weakens. Separately, retrain the PRS model with verbatim rationale spans instead of generated ones and check whether the complex-problem advantage persists.","supporting_citations":[{"cited_title":"2019 , publisher =","cited_arxiv_id":null,"evidence_quote":"SciBERT provides the embeddings used in BERTScore for span matching and rationale evaluation."}],"review_version":1}