{"id":"5c975c27-f8e2-46c4-ac71-507a5f6ca737","arxiv_id":"2508.20374","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TCIA augments task-specific instruction data by decomposing instructions into queries and constraints and using retrieval-guided breadth-first search to generate on-task variants, improving fine-tuned Llama-3.1-8B on four proprietary meeting tasks without a measured drop in general ability.","lead":"This paper introduces TCIA, a way to automatically expand the instructions used to train a language model for a specific job by splitting each instruction into a core request plus constraints, then borrowing constraints from similar tasks. The authors report average gains of 8.7% on four internal meeting summarization and extraction tasks while keeping scores on general benchmarks roughly unchanged.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 8.7% gain is measured only on four undisclosed in-house tasks with no code, data, or uncertainty; until a public replication or release of evaluation materials exists, the central empirical claim is unverifiable.","rationale":"The reader's named weakest assumption is LLM decomposition fidelity. That is a real mechanism-level risk, but it is not the most load-bearing issue for the central claim: even if decomposition were perfect, the 8.7% headline would remain unsupported because all four evaluation tasks are hidden and the evaluation protocol is partly LLM-as-judge. The empirical claim is the paper's main contribution, and it cannot be checked from the manuscript. The internal comparisons are properly controlled in design, which is why I do not move the verdict to REJECT: the conditional verdict is the right default. I set agreement_with_reader to partial because the reader's rationale already flags missing code, data, and uncertainty, but the named weakest assumption is not the same as my primary concern. A public replication or release of one task's evaluation materials would convert the concern into a checkable result.","tokens_in":19474,"tokens_out":9818,"duration_ms":92136,"concrete_test":"Release the evaluation data, judge prompts, and per-run scores for at least one of the four in-house tasks, then have an independent evaluator recompute the TCIA-vs-FI and TCIA-vs-WizardLM gaps. Alternatively, run the same SFT comparison on a public meeting-summarization benchmark (e.g., MeetingBank) using the Table 16 seed prompt and identical hyperparameters. If the reproduced average gap does not approximate the reported 8.7% improvement over FI and 3% over WizardLM, the headline result is not robust to the undisclosed evaluation setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Figure 3) is that TCIA improves open-source LLMs by an average of 8.7% across four real-world, task-specific applications and surpasses GPT-4o. Every piece of evidence for this claim comes from four proprietary meeting-AI tasks (Task A-D) that the authors explicitly state cannot be disclosed due to company policy (Experiments, footnote 2). The paper does not release the evaluation data, task prompts beyond one illustrative seed, the exact LLM-as-judge prompts, per-run scores, or code. Because the reference-free evaluation uses variants of the LLM-as-a-Judge protocol, and the SFT data themselves were generated and filtered by the same class of LLMs (claude-3-5-sonnet, gpt-4o, gpt-4.1), the reported advantage could reflect judge preference for LLM-generated instruction and response styles rather than task-level capability. The only internal evidence visible to a reviewer is Table 3 and Figure 3, neither of which can be independently checked. The no-sacrifice claim is also muddled: Table 4 shows TCIA's 51.33 average is below Llama-8B's 54.67 and Tulu-8B-SFT's 51.99, while the prose says TCIA matches FI; the general-benchmark support is weaker than claimed. These weaknesses are addressable if materials are released, but as it stands the central empirical claim is not independently verifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TCIA, a six-stage instruction augmentation pipeline: LLM-based decomposition of instructions into base queries and categorized constraints; construction of a task-organized instruction database from Tulu-3; breadth-first search over constraint sets using Add, Remove, and Replace operations; conversion of the augmented states back into natural-language instructions; LLM-based validation; and multi-LLM response generation with quality filtering. The resulting data are used for supervised fine-tuning of Llama-3.1-8B. The authors report that TCIA sustains instruction diversity and on-task ratio across three hops, achieves higher constraint pass rates than Fixed Instruction and WizardLM on unseen constraints (Table 3), improves average end-to-end scores by 8.7% over FI and 3% over WizardLM on four proprietary meeting-AI tasks while surpassing GPT-4o (Figure 3), and preserves general benchmark performance (Table 4).","tokens_in":19737,"tokens_out":5343,"duration_ms":49403,"significance":"The controlled three-way SFT comparison (FI, WizardLM, TCIA) under identical hyperparameters, generation pipelines, and filtering is methodologically sound, and the BFS state-space formulation is clearly specified. The prompt-level diversity and on-task analyses in Figure 1 and Figures 4-5 provide a concrete mechanism for TCIA's claimed benefits, and the paper is explicit about its experimental setup and several prompts. If the empirical claims hold, TCIA is a practically relevant contribution to task-specific instruction tuning. However, the headline results rest entirely on four undisclosed in-house tasks with no released data, code, judge prompts, or uncertainty quantification, and the general-benchmark 'no sacrifice' claim is only weakly supported by the reported numbers. The strengths do not yet outweigh the verification gap.","major_comments":[{"comment":"The central empirical claim---an average 8.7% improvement over FI and superiority over GPT-4o---is not independently verifiable. Footnote 2 states that the four in-house tasks A-D cannot be disclosed due to company policy, and no code, evaluation data, task prompts, LLM-as-judge prompts, per-run scores, or variance estimates are provided; Figure 3 reports only averages of three runs. The paper should either release sanitized or anonymized task specifications and the full evaluation protocol, or include a public replication study on open tasks; without one of these, the headline result cannot be checked by readers.","section":"Experiments; Figure 3; footnote 2"},{"comment":"The quality filtering and reference-free evaluation share the same model family, creating a self-preference confound. SFT data are generated by claude-3.5-sonnet, gpt-4o, and gpt-4.1 and filtered by gpt-4.1 (Table 17), while the reference-free evaluations use unspecified 'variants of the LLM-as-a-Judge protocol' with no judge model named. If the evaluation judge is also gpt-4.1 or gpt-4o, the reported gains may partly reflect stylistic preference for data generated and filtered by those models rather than task-level capability. Please specify the judge models, report agreement with the human-annotated subset, and include at least one hold-out judge from a different model family.","section":"Experiments: SFT Setup and Baselines; Data Quality Filtering; Table 17"},{"comment":"The no-sacrifice claim is not supported by the reported numbers. TCIA's average of 51.33 is below Llama-8B (54.67) and Tulu-8B-SFT (51.99), and the sentence 'matches the average benchmark score of FI (50.17 vs. 51.33)' is internally inconsistent because TCIA actually exceeds FI by 1.16 points, while the two reference models score higher. Similarly, 'matches or exceeds Tulu-8B-SFT on almost all public benchmarks' is not accurate on IFEval, GPQA, BBH, and the overall average. Please correct these claims and add a more careful comparison, ideally with significance testing or at least variance reporting.","section":"Generalization on Public LLM Benchmarks; Table 4"},{"comment":"The framework assumes that the LLM decomposition returns a complete and faithful query-constraint representation, but the paper reports no accuracy, recall, or human-agreement evaluation for this step. Since every Add and Replace operation mutates the extracted constraints, a missing or mislabeled constraint is inherited and amplified through the BFS; the on-task ratios in Figure 1d and the downstream gains in Figure 3 could therefore be specific to the decomposition model rather than to TCIA itself. Please include a human-annotated decomposition evaluation on a sample from Tulu-3 and from the seed tasks, reporting per-category constraint recall.","section":"Instruction State Decomposition; BFS for Instruction Augmentation"},{"comment":"The method has four free parameters (K, m, k, and the number of BFS hops), yet the paper reports no sensitivity analysis, no ablation of the individual BFS operations, and no comparison against a simple random constraint-sampling baseline. The reported gains could depend on these unexamined choices. Please add ablations that vary k and the number of hops and that replace BFS with a non-retrieval random-add baseline, to establish which components of TCIA are actually responsible for the improvements.","section":"Table 17; Algorithm 1"}],"minor_comments":[{"comment":"There is a stray space in 'inputs .' in the first sentence that should be corrected.","section":"Abstract"},{"comment":"The phrase 'such as like WizardLM' contains a duplicated comparative marker and should read 'such as WizardLM'.","section":"Related Works"},{"comment":"Figure 3 reports averages of three runs but shows no error bars, ranges, or per-run points; please add at least standard deviations or min-max ranges so the reader can judge stability.","section":"Figure 3"},{"comment":"The LLM used for instruction decomposition in the main pipeline is not stated in the main text; it appears only in the appendix via Table 18 (GPT-4.1). Please state it explicitly in the setup.","section":"Experiments: SFT Setup and Baselines"},{"comment":"The claim of being 'the first task-centric instruction augmentation framework' is not substantiated with a comparison to prior task-specific or retrieval-augmented instruction generation methods; consider softening the claim.","section":"Conclusion and Contributions"},{"comment":"In the Data Scoring - Uncertainty and Data Scoring - Truthfulness prompts, the literal text '\\None' appears where a numeric identifier is intended; this is likely a LaTeX artifact and should be fixed.","section":"Appendix: Data Scoring prompts"}],"recommendation":"major_revision","confidential_remarks":"This is an industry paper whose central evaluation is entirely on proprietary tasks, which creates a reproducibility gap that the program committee may want to weigh against the conference's data-availability norms. The internal inconsistency in the general-benchmark prose around Table 4 should be treated as a substantive correctness issue, not merely a wording problem, because it directly supports the 'no sacrifice' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TCIA is a coherent pipeline for task-specific instruction augmentation, and the three-way SFT comparison is the right experiment. The problem is that the headline 8.7% gain is measured only on four proprietary meeting tasks with no released data, code, or uncertainty. Right now the central empirical claim is unverifiable.\n\nWhat is actually new is the specific combination: decompose instructions into query plus typed constraints, retrieve similar constraints from Tulu-3, run BFS with Add/Remove/Replace, convert back, validate, and filter with LLM judges. Each component is known, but the task-centric retrieval and the constraint-state BFS are a reasonable synthesis. The internal comparison is well-designed: FI, WizardLM, and TCIA share hyperparameters, generation model pool, and filtering, so the setup does not obviously favor TCIA. Table 3 is also a concrete plus: on Task A, TCIA-8B shows much higher pass rates on unseen constraints than FI-8B and WizardLM-8B.\n\nNow the soft spots, in order of severity. First, evaluation transparency. The four tasks are undisclosed, and no code, data, or judge prompts are released. That makes the main claim impossible to check. There are no error bars or significance tests despite three runs. Second, the judge overlap: gpt-4.1 is used for response generation, data filtering, and the reference-free evaluation, so some of the reported gain may be judge preference for LLM-generated styles rather than task-level capability. Third, the prose does not match the tables. The 'matches or exceeds Tulu-8B-SFT on almost all public benchmarks' claim is wrong: TCIA is below Tulu on IFEval, GPQA, and BBH, and its average (51.33) is below Tulu's (51.99). The end-to-end numbers are also inconsistent: the task-wise improvements over FI mix absolute and relative percentages, and TCIA ties GPT-4o on Task C despite the claim that it 'exceeds' GPT-4o on all tasks. Fourth, the decomposition is the load-bearing component; if the LLM decomposer is noisy, the BFS mutations inherit and amplify that noise. That is an addressable weakness, not a refutation.\n\nWho this is for: practitioners who want a concrete recipe for building task-specific instruction sets. The method is plausible and the controlled comparison is a good start. A serious referee should see it, but the current version needs major revision. I would send it to peer review with the expectation that the authors release the evaluation materials, report variance, fix the benchmark and end-to-end prose, and add at least one public task to anchor the claims.","headline":"A plausible task-centric instruction augmentation pipeline, but the headline 8.7% gain is built on undisclosed proprietary data and inconsistent prose; worth peer review only if materials are released.","tokens_in":20348,"tokens_out":6123,"would_cite":false,"duration_ms":50013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that instruction augmentation succeeds when it is constrained to stay on-task, and shows that its TCIA framework raises open-source LLM performance by 8.7% on average across four real-world tasks without sacrificing…","keywords":["instruction tuning","data augmentation","task-centric instruction generation","constraint decomposition","breadth-first search","instruction following","synthetic instruction data","LLM fine-tuning"],"falsifier":"Run TCIA on a held-out task with two decomposers that differ measurably in extraction accuracy (or with the same decomposer but with 10% of constraints deliberately dropped), then compare on-task ratios and downstream pass rates; if they do not fall when decomposition quality falls, the claim does not depend on decomposition fidelity, but if they track it, the framework's gains are bounded by the decomposer.","tokens_in":19243,"feed_emoji":"🎯","tokens_out":6760,"duration_ms":58059,"temperature":0.7,"pith_summary":"The paper claims that instruction augmentation for fine-tuning large language models fails in practice not for lack of diversity but for loss of task relevance, and that this can be fixed by treating each instruction as a base query plus a set of explicit constraints. The proposed framework, TCIA, decomposes seed instructions into that discrete form, retrieves similar constraints from a task-organized instruction database, and explores new constraint combinations with a breadth-first search. Fine-tuning Llama-3.1-8B on the resulting instruction–response pairs raised scores by an average of 8.7% over a fixed-instruction baseline across four proprietary meeting tasks, and by about 3% over WizardLM-augmented data, even exceeding GPT-4o on all four tasks. The paper also reports that the same models keep their general instruction-following scores on public benchmarks, indicating specialization did not consume generality. A sympathetic reader would take this as evidence that constraint-centric, on-task augmentation is a scalable route to task-specific LLMs.","feed_headline":"Task-centric instruction data lifts LLM scores 8.7%","feed_subtitle":"TCIA keeps generated instructions on-task across three hops, beating WizardLM and GPT-4o on specialized meeting tasks.","key_machinery":"The load-bearing mechanism is the query-constraints state representation: an instruction is rewritten as a base query Q plus a categorized constraint set C (content, numerical, style/tone, format, language, input placeholder), with a task type T assigned. The framework's breadth-first search then treats constraint sets as states and applies three operations—Add (insert a constraint retrieved from a similar task), Remove (delete one), and Replace (swap in a semantically similar constraint)—guided by embedding-based retrieval over a task-organized instruction database built from Tulu-3. The natural-language reconstruction step with critique-and-refine, the validation step, and the five-dimension LLM-as-a-judge filtering are quality gates that keep the generated data usable for SFT; the core identity is that instruction diversity can be generated by composing and mutating constraints while keeping Q and T fixed.","core_discovery":"The central discovery, stated in the paper's own terms, is that task drift—not diversity—is the binding constraint on automatic instruction augmentation, and that a discrete query-constraints state space makes task relevance controllable. TCIA starts from one seed instruction per task, extracts the base query Q, task type T, and constraint set C, then runs a breadth-first search with Add, Remove, and Replace operations that pull candidate constraints from semantically similar tasks in an instruction database built from Tulu-3. Each mutated state is converted back to natural language with an LLM, verified for constraint coverage, validated for feasibility, paired with task-specific context, and filtered by an LLM-as-a-judge before supervised fine-tuning. The paper reports that this pipeline sustains near-100% on-task ratios across three augmentation hops while WizardLM's on-task ratio falls below 60%, and that the resulting TCIA-8B model achieves higher pass rates on unseen constraints and higher end-to-end scores than FI-8B, WizardLM-8B, and GPT-4o on all four proprietary tasks.","pith_inferences":["An implication the paper leaves implicit is that the same constraint-mutation machinery could be turned into a controllable stress-test generator: by targeting specific constraint categories (format, length, tone), one could deliberately probe which instruction-following skills a model lacks.","The near-100% on-task ratio suggests TCIA could also serve as a data-centric diagnostic, meaning that measuring an augmentation method's on-task ratio before training may predict downstream constraint robustness—a testable proxy the paper does not itself run.","Because the constraint pool comes from Tulu-3, the framework's ceiling likely depends on the breadth and quality of the retrieval database; a natural extension would be to measure how downstream gains scale as the database grows or as retrieved constraints come from the target domain itself."],"forward_implications":["One seed instruction per task is enough to produce a high-quality SFT dataset of about 10k filtered instruction–input pairs per task.","Models trained on TCIA data follow unseen user constraints (numbered lists, length limits, entity groupings) substantially better than fixed-instruction or WizardLM-augmented models.","Task-specific fine-tuning with TCIA does not trade away general capability: averaged public-benchmark scores stay near the fixed-instruction baseline and close to Tulu-8B-SFT.","Open-source 8B models fine-tuned on TCIA data can exceed a leading closed model such as GPT-4o on specialized, production-oriented tasks, suggesting the gap is data distribution rather than model scale.","The framework is a pipeline that can be rerun for any new task by supplying a seed prompt and task context, so its benefits should transfer to other verticals as long as the instruction database has related constraints."],"supporting_citations":[{"why":"WizardLM is the main augmentation baseline whose diversity collapse and task drift TCIA is designed to fix, and whose generation setup (3 hops, 2k instructions) TCIA follows for fair comparison.","marker":"Xu et al. 2024"},{"why":"Tulu-3 supplies the broad instruction database used for constraint retrieval and is also the source of the five-dimension quality criteria used in data filtering.","marker":"Lambert et al. 2024"},{"why":"Conifer provides the prior structured decomposition of instructions into base queries and constraints that motivates the state representation adopted by TCIA.","marker":"Sun et al. 2024"},{"why":"UltraIF is cited as supporting the decomposition-based approach to instruction following, grounding TCIA's choice to extract task types and explicit constraints.","marker":"An et al. 2025"},{"why":"Llama-3.1 is the base model for all supervised fine-tuning runs and also provides the Llama-8B-Instruct baseline for comparison.","marker":"Grattafiori et al. 2024"},{"why":"GPT-4o serves as the strong closed-source baseline that TCIA-8B must beat, and it is also one of the LLMs used to generate candidate responses.","marker":"OpenAI 2024"}],"fun_headline_variants":["TCIA keeps instruction tuning on-task, beats GPT-4o","Task-centric instruction augmentation lifts LLM performance 8.7%","On-task instruction data: TCIA surpasses GPT-4o","TCIA controls task drift with discrete query-constraints space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's decomposition of each instruction into a base query plus a complete, correctly categorized set of constraints is faithful; if a constraint is missed, mislabeled, or invented, the BFS mutation and retrieval propagate that error into most generated instructions.","fun_headline_variants_meta":{"raw":{"variants":["TCIA keeps instruction tuning on-task, beats GPT-4o","Task-centric instruction augmentation lifts LLM performance 8.7%","On-task instruction data: TCIA surpasses GPT-4o","TCIA controls task drift with discrete query-constraints space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2901,"prompt_tokens":1014,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":1813}},"tokens_in":630,"tokens_out":1887,"duration_ms":12288,"temperature":1.0,"reasoning_tokens":1813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:46:32.610439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TCIA on a held-out task with two decomposers that differ measurably in extraction accuracy (or with the same decomposer but with 10% of constraints deliberately dropped), then compare on-task ratios and downstream pass rates; if they do not fall when decomposition quality falls, the claim does not depend on decomposition fidelity, but if they track it, the framework's gains are bounded by the decomposer.","supporting_citations":[],"review_version":2}