{"id":"8c4bd567-6654-457a-b266-1a8be8c37a36","arxiv_id":"2607.27379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.","lead":"This paper builds a three-stage pipeline that mines humanities/social-science texts from the web, turns them into instruction-answer pairs, and uses those pairs to fine-tune language models. The resulting 230k-sample dataset beats 14 existing synthetic-data methods on 16 benchmarks for the Qwen3-8B base model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'human-preference' SOTA is not established: data quality is graded by Qwen, final 'human preference' by Claude LLM judges, with no human evaluation of model outputs; length/style bias may drive the gains.","rationale":"I read the paper as a serious, transferable data-synthesis pipeline with broad benchmark coverage and honest limitations. The strongest claim, however, is not just that the data is internally rated high-quality, but that it improves human preference without sacrificing knowledge. That claim is load-bearing because the headline SOTA is driven by the human-preference columns. The reader's weakest assumption identified the self-referential quality controls inside the pipeline; I agree with that, but I would sharpen it: the final 'human-preference' evaluation is itself performed by LLM judges, so the loop is not closed with human judgment anywhere on the final outputs. This is a construct-validity problem, not an internal contradiction or fraud allegation. The paper's transfer results and ablations are useful, and the knowledge-benchmark improvements are plausible, but the 'human-preference SOTA' and 'approaches official Qwen3-8B' claims should be conditioned on a human evaluation. Therefore I do not change the reader's CONDITIONAL verdict; I would simply make the condition more precise: independent human preference testing, length-controlled, plus uncertainty quantification, before the central claim is taken at face value.","tokens_in":29919,"tokens_out":5450,"duration_ms":58384,"concrete_test":"Run a human pairwise preference study on sampled outputs from HSS-Synth, the strongest baseline (e.g., WebR-Pro), and the official Qwen3-8B, using WritingBench and EQ-Bench prompts. Use expert/domain raters, blind to system, with and without length-matching (e.g., truncating longer responses to the shorter response's length). If HSS-Synth does not win significantly on human preference (e.g., McNemar test, p<0.05), or wins only under length-uncontrolled presentation, the core 'human-preference without performance seesaw' claim is an LLM-judge artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HSS-Synth improves 'human preference' without a performance seesaw (Table 1) rests on benchmarks whose human-preference components are scored by LLM judges: WritingBench, CreativeWriting-v3, and Judgemark use Claude-3.7-Sonnet; BuzzBench and EQ-Bench3 use Claude-3.5/3.7-Sonnet (§4.1). Meanwhile, the entire quality-control stack — domain classification, 12-rubric quality rating, refinement judging, Q&A alignment checking, and teacher-forced answering — is performed by Qwen3-30B-A3B (§3.1–§3.3). The rubric validation in §4.6 is limited to 400+ seed documents with human raters; it does not validate the final model outputs. The pipeline may therefore be optimizing answers to what LLM judges reward, and the benchmark 'human preference' scores may reflect LLM-judge preferences rather than human preferences. This is especially plausible because HSS-Synth's largest gains over the best baseline are in Writing (+10.5) and Emotion (+6.0), exactly the LLM-judged rubric tasks. The appendix's own case (Table 10) shows the teacher-forced answer is 644 tokens against an explicit 413-token instruction constraint, yet is presented as the best answer type — suggesting the LLM judge rewards length/detail over instruction fidelity. No human preference study on model outputs, no significance tests, and no length-controlled evaluation are reported, so the 'human-preference SOTA' is not yet externally validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HSS-Synth, a three-stage pipeline for synthesizing instruction-tuning data in 14 humanities and social science domains. Stage 1 constructs seed documents from SlimPajama via source sampling, heuristic filtering, LLM domain classification, a 12-rubric quality rating, and LLM text refinement with a refinement judge. Stage 2 performs multi-attribute instruction backtranslation (requirements plus persona) with a Q&A alignment check. Stage 3 proposes teacher-forced answering (TeachForceA), where the seed document is fed alongside the instruction as a semantic anchor. The authors fine-tune Qwen3-8B-Base on the resulting ~230k samples and report improvements over 14 baselines across 16 benchmarks, claiming a new SOTA that approaches the official Qwen3-8B while avoiding a 'performance seesaw' between human preference and knowledge capabilities. They also report ablations, cross-model transfer to Llama3.1-8B and Qwen2.5-14B, scaling analyses, and a rubric validation study.","tokens_in":30270,"tokens_out":3156,"duration_ms":32338,"significance":"If the central claims hold, the work is a substantial step for open-ended, subject-centric data synthesis: it defines a principled HSS taxonomy, offers a reproducible pipeline, ships public code, and provides broad benchmark coverage with a decontamination check. The cross-architecture transfer and component ablations are also useful. However, the paper's signature claims are currently supported by evidence that is partly circular and partly indirect. The human-preference benchmarks are judged by Claude LLMs rather than humans; the quality-control stack is run entirely by the same model family (Qwen3-30B-A3B) that generates the data; rubric validation is limited to 400+ seed texts, not final model outputs; and no significance tests, multiple seeds, or length-controlled evaluations are reported. These gaps directly affect the headline 'SOTA' and 'no performance seesaw' claims, so the manuscript requires substantive additional validation before the claims can be accepted.","major_comments":[{"comment":"The 'human preference' category is not measured with humans. WritingBench, CreativeWriting-v3, Judgemark, BuzzBench, and EQ-Bench3 are all scored by Claude-3.5/3.7-Sonnet (the paper states this in §4.1). The claim of 'improving human preference' is therefore a claim about LLM-judge preference. No human preference study on the final model outputs is reported. Please add (a) a human evaluation of the fine-tuned models' outputs, or (b) at minimum a length-controlled LLM-judge evaluation (e.g., following Dubois et al., 2024, already cited), and (c) variance or significance measures across multiple seeds, since Table 1 reports point estimates with no error bars.","section":"§4.1 / Table 1"},{"comment":"The pipeline uses Qwen3-30B-A3B for domain classification, quality rating, text refinement, refinement judging, Q&A alignment checking, and teacher-forced answering. The rubric validation in §4.6 is only on 400+ seed documents with human raters and a GPT-4.1 comparison; it does not validate the quality of the final instruction–answer pairs. This creates a circularity risk: the dataset may be optimized toward what the synthesis model (and its relatives) reward, rather than toward genuine HSS quality. Please validate the final dataset and the trained models with independent judges, ideally including human ratings of a sample of final (instruction, answer) pairs.","section":"§3.1–§3.3 / §4.6"},{"comment":"The abstract claims 237k samples, §4.1 says 230k instruction–answer pairs were used for fine-tuning, and Appendix C reports 237,340 total samples. This discrepancy is not explained. If 230k is the training-set size, the abstract and appendix should be reconciled; if the dataset is larger than the training split, the filtering/selection procedure should be described. As written, the reader cannot determine what was actually trained on.","section":"Abstract / §4.1 / Appendix C"},{"comment":"The case study contradicts the instruction-fidelity claim. The reverse instruction explicitly requires 'approximately 413 tokens (around 2130 characters)', but the reverse answer is 433 tokens and the teacher-forced answer is 644 tokens, with the latter praised as satisfying the length constraint. This suggests the LLM judge used for rubric scores rewards length/detail over instruction fidelity, and it directly undermines the claimed advantage of TeachForceA. Please provide a length-controlled evaluation of answer quality and report the distribution of output lengths relative to instruction-specified lengths.","section":"Appendix E / Table 10"},{"comment":"The 'no performance seesaw' and 'no cross-skill trade-offs' conclusions are based on comparing averages over heterogeneous benchmarks with different metrics and no confidence intervals. The scaling curves in Figure 2 are also point estimates. Without error bars, significance tests, or multiple seeds, the absence of a trade-off is not established. Please report per-seed variation or bootstrap confidence intervals for the key comparisons in Table 1 and Figure 2.","section":"§4.2 / Figure 2"}],"minor_comments":[{"comment":"Typo: 'Evlo-instruct' should be 'Evol-Instruct'. Also, Table 1 contains a run-together entry '51.2937.9053.36' in the SynthQuestions row; please fix the formatting.","section":"Table 3"},{"comment":"Captions: Figure 5's caption says 'model trained on seed documents' and Figure 6 says 'model trained on reverse answers', but the final sentence of Figure 6's caption reads 'The model trained on seed answers clearly finds its answer pattern easy to learn', which appears to be a copy-paste error (should be 'reverse answers').","section":"Figure 5 / Figure 6"},{"comment":"Minor nomenclature: 'Slimpajama' should be 'SlimPajama'. In Table 1, baseline rows list 'Qwen-30B-A3B' for WRAP and LongForm while the text uses 'Qwen3-30B-A3B'; please harmonize.","section":"§3.1"},{"comment":"The abstract phrase 'teacher-forced Answering that fed seed documents during response' is grammatically unclear; suggest 'during response generation' or similar. Also, the term 'Teacher-Forced Answering' is introduced in §1, §3.3, and again in Table 1; please define it once in a glossary-like manner.","section":"Abstract / §3.3"},{"comment":"The Cosmopedia-v2 reference lists only 'Loubna Ben, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, Leandro von Werra' and the year; if this is the Cosmopedia technical report, please give the full author list or a citable handle. Some other entries (e.g., 'Tombench') also mix arXiv identifiers with incomplete author lists.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely to be of interest to a CL/NLP audience, but the central empirical claims are not yet supported by the current evidence. The circularity concern (same model family for generation and judging) and the use of LLM judges for the 'human-preference' benchmarks are not merely presentation issues; they are load-bearing for the claimed SOTA. I recommend requesting the additional validation outlined in the major comments before acceptance. I would also ask the editor to ensure that the released code and data (claimed to be public) are actually made available for verification during revision, since the paper's reproducibility hinges on the exact pipeline thresholds and the training–evaluation split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful data-synthesis recipe for an underserved domain, and the experiments are more thorough than most papers in this space. But the SOTA/human-preference headline outruns the evidence: the preference benchmarks are LLM-judged, the pipeline's quality checks use the same model family, and there is no human evaluation of final outputs. The method deserves a serious referee, not because the headline is proven, but because the work is solid enough to be worth tightening.\n\nWhat's new: the subject-centric 14-field HSS taxonomy, the specific combination of multi-attribute backtranslation with Q&A alignment checking, and teacher-forced answering. Individually those parts exist (LongForm, WebR, WRAP), but the integration and the HSS focus are new. What's done well: they compare against 14 baselines, including re-running WRAP and LongForm under the same seed data and synthesis model, which is the right way to isolate pipeline contribution. Transfer to Llama-3.1-8B and Qwen2.5-14B, ablations, data-quality analysis, and a 400-document human rubric validation. The rubric validation is a real plus, even if it only covers seed texts.\n\nSoft spots. The 'human preference' evaluation is not human. WritingBench, CreativeWriting, Judgemark, BuzzBench, and EQ-Bench are all Claude-graded; the paper calls them human-preference but no human raters scored model outputs. The pipeline's domain classification, quality rating, refinement judging, Q&A alignment checking, and answer generation all use Qwen3-30B-A3B. The 86% human-agreement check is on seed texts, not final answers. So the data could be optimized to what Qwen and Claude judges reward. The Table 10 case makes this concrete: the teacher-forced answer is 644 tokens against an explicit 413-token instruction, yet the paper presents it as the best answer type. That looks like length/detail bias in the judge, not genuine instruction fidelity. No error bars or seeds in Tables 1–2, dataset not released, and the abstract's 237k vs 230k used for training is a small but avoidable inconsistency. These don't sink the method, but they prevent taking the SOTA claim at face value.\n\nWho it's for: people building instruction-tuning datasets for open-ended domains. It's a reasonable template and a good baseline for future HSS synthesis work.\n\nRecommendation: send to peer review. The experiment suite is substantial and the limitations are addressable; a referee can ask for human eval of a sample of outputs, length-controlled judgment, and multiple seeds. That path is more useful than desk-rejecting.","headline":"Solid method for HSS data synthesis, but the human-preference SOTA is built on LLM judges and needs independent validation.","tokens_in":30806,"tokens_out":2607,"would_cite":true,"duration_ms":24975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Web text becomes 237k HSS instruction pairs; teacher-forced answers lift a fine-tuned 8B model past 14 rival datasets.","keywords":["data synthesis","instruction tuning","humanities and social sciences","teacher-forced answering","backtranslation","Q&A alignment","web corpora","fine-tuning"],"falsifier":"Take a random sample of 1,000 HSS-Synth instruction–answer pairs and have independent human experts (not the paper's raters) rate them against the same 12 rubrics and a Q&A faithfulness check; or re-run the whole pipeline using a different model family (e.g., a non-Qwen judge) for quality rating and Q&A alignment and then compare downstream fine-tuning performance. If the quality advantage shrinks or vanishes, or if the human experts find systematic misalignment, the pipeline's internal LLM-based validation is circular and the headline gains are artifacts of self-preference.","tokens_in":29788,"feed_emoji":"📚","tokens_out":4830,"duration_ms":43186,"temperature":0.7,"pith_summary":"The paper sets out to show that high-quality, diverse instruction-tuning data for open-ended humanities and social-sciences tasks can be synthesized automatically from web corpora, rather than collected from humans or distilled from a frontier model. It builds a three-stage pipeline that filters and refines seed documents, backtranslates them into persona- and requirement-specific instructions, and anchors LLM answers to the seed documents during generation. If the central claim is right, the approach removes a major data bottleneck for HSS domains and transfers across model architectures and sizes, with a fine-tuned Qwen3-8B approaching the official instruction-tuned release.","feed_headline":"Web text yields 237k HSS pairs; fine-tuned 8B beats 14 datasets","feed_subtitle":"Automated pipeline with teacher-forced answers pushes fine-tuned 8B close to official Qwen3-8B.","key_machinery":"The load-bearing machinery is the three-stage pipeline. Stage one builds clean seed documents from web text through source sampling, heuristic filtering, LLM domain classification into 14 HSS fields, 12-rubric quality rating (readability, applicability, human-touch), LLM text refinement, and a refinement judge. Stage two is multi-attribute instruction backtranslation: the LLM writes instructions that specify domain, genre, length, core content, structure, narrative voice, and a persona ('what' and 'who'), then a Q&A alignment check confirms the reverse instruction can faithfully reproduce the seed. Stage three is TeachForceA, which feeds the seed document to the answering model as a semantic","core_discovery":"The central discovery is teacher-forced answering (TeachForceA): when generating an answer, the synthesis model is given the seed document alongside the instruction as a semantic anchor. This reduces hallucination, preserves the original tone and style, and improves completeness and readability, producing answers that score higher on the paper's 12-rubric quality scale and lead to better fine-tuned models than either the raw seed document or a purely reverse-generated answer. Combined with multi-attribute instruction backtranslation (which specifies both the task requirements and the persona) and a Q&A alignment check, the pipeline yields 237k instruction–answer pairs from Qwen3-30B-A3B that","pith_inferences":["The same teacher-forced anchoring idea could apply to other open-ended synthetic-data pipelines (e.g., medicine, law, advice-giving), where grounding answers in a source document may reduce hallucination and style drift without human annotation.","Because the same Qwen model family both generates and judges the data, the reported 'quality' may partly reflect the synthesis model's own preferences; a direct test would be to re-run the pipeline with an independent judge model or human raters and see whether downstream gains survive.","The observed saturation pattern suggests that SFT injects little new factual knowledge; this implies that for knowledge-heavy HSS applications, larger gains would come from synthesizing pre-training data rather than instruction data.","The paper's reuse metrics (n-gram overlap, LCS, copy ratio) offer a cheap, general test for grounding in any synthetic QA pipeline; a low copy ratio with high quality could indicate genuine abstraction rather than copying."],"forward_implications":["Fine-tuning a base model on HSS-Synth outperforms 14 leading instruction-data baselines across 16 benchmarks, coming closest to the official instruction-tuned Qwen3-8B.","Teacher-forced answers are the best answer type; raw seed documents are unsuitable as answers because their embedding distribution diverges sharply from reverse-generated answers.","The gains transfer across model families and sizes: Qwen2.5-14B and Llama3.1-8B both improve markedly on HSS-Synth data.","There is no detected 'performance seesaw': human-preference gains do not come at the cost of knowledge benchmarks, and some knowledge capabilities improve slightly.","Different capabilities saturate at different data scales: knowledge-based skills converge around 10% of the 237k samples, human-preference skills around 20%, and instruction following around 33%."],"fun_headline_variants":["Teacher-forced answers anchor LLM data synthesis","237k HSS pairs via teacher-forced answering","Fine-tuned 8B nears Qwen3-8B with teacher forcing","HSS-Synth: seed anchors beat 14 datasets","Teacher-forcing yields 237k HSS instruction pairs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pipeline's quality controls—domain classification, rubric rating, text refinement, refinement judging, Q&A alignment, and teacher-forced answering—are all performed by the same LLM family (Qwen3-30B-A3B) that generates the data, and the rubric's validity rests on only 400+ seed texts rated by human annotators plus a single GPT-4.1 comparison; if these judgments are biased or self-referential, the dataset may be optimized to the synthesizer's taste rather than to true HSS","fun_headline_variants_meta":{"raw":{"variants":["Teacher-forced answers anchor LLM data synthesis","237k HSS pairs via teacher-forced answering","Fine-tuned 8B nears Qwen3-8B with teacher forcing","HSS-Synth: seed anchors beat 14 datasets","Teacher-forcing yields 237k HSS instruction pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1787,"prompt_tokens":795,"completion_tokens":992,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":908}},"tokens_in":539,"tokens_out":992,"duration_ms":8410,"temperature":1.0,"reasoning_tokens":908,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:25:26.283270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 1,000 HSS-Synth instruction–answer pairs and have independent human experts (not the paper's raters) rate them against the same 12 rubrics and a Q&A faithfulness check; or re-run the whole pipeline using a different model family (e.g., a non-Qwen judge) for quality rating and Q&A alignment and then compare downstream fine-tuning performance. If the quality advantage shrinks or vanishes, or if the human experts find systematic misalignment, the pipeline's internal LLM-based validation is circular and the headline gains are artifacts of self-preference.","supporting_citations":[],"review_version":1}