{"id":"d92513d4-72cc-4fcf-9687-bea8a432e6b8","arxiv_id":"2608.10319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On 206 real developer-agent sessions, personalized skills distilled from a developer's own history give small inconsistent gains, while pooled generic skills show the largest (marginally significant) improvement.","lead":"This paper tests whether coding agents improve when they are given a personalized instruction sheet learned from a developer's past work. It finds the personalized instruction barely helps, while a generic instruction pooled across many developers gives the largest, though not statistically significant, improvement.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generic-skill condition leaks the target developer's own evolution sessions into the pooled skill (Section 3.2), so the headline 'shared knowledge beats personalization' is confounded with the generic skill simply being a superset of the personalized skill.","rationale":"The paper asks a real question and the replay framework is a reasonable attempt at controlled comparison. The negative finding about personalized skills is supported by the numbers: B vs A has p=.399 and a win rate below 50%, and B vs C shows no personalization benefit. The weak spot is the positive generic claim. The generic condition has an information advantage by construction: it pools all developers' evolution sessions, including the target's. This is stated openly in Section 3.2, but the interpretation in the abstract and Section 6 overreaches. Because the generic skill is a superset of the personalized skill, the observed +3.78 could be due to larger skill size, broader coverage, or simply to the inclusion of the target's own traces, not to sharedness. The lack of a leave-one-out control means the comparison cannot distinguish 'shared knowledge helps' from 'more data helps.' The reader's simulator-equivalence concern is real, but it affects all conditions symmetrically; the leakage concern directly biases the key B-vs-D comparison and the central conclusion. A single leave-one-out replay experiment would settle this. Given this, the conditional verdict is appropriate; no verdict change is needed, but the required condition should include this control.","tokens_in":15137,"tokens_out":5258,"duration_ms":55880,"concrete_test":"Re-run Condition D as a leave-one-out generic skill: for each developer u, build the generic skill from the pooled evolution sessions of the other 12 developers only, excluding u, and replay the same 210 test instances (42 sessions x 5 seeds) with all other components unchanged. Compare the leave-one-out generic score, follow-up rate, and win/tie/loss against the no-skill baseline and against Condition B. If the leave-one-out generic improvement over baseline drops to the personalized-skill level (around +1) or below, the claim that shared knowledge is more robust is not supported; if it retains a gain near +3.78, the leakage concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Condition D is constructed from the pooled evolution sessions of all developers, explicitly including those of the target developer (Section 3.2). The personalized skill B uses only the target developer's sessions. D therefore contains all the information in B plus additional instructions from other developers, making D a strict superset of B. Under this design, the finding that D outperforms B (Table 1: 68.80 vs 65.99) does not distinguish between two very different explanations: (i) broadly shared procedural knowledge is genuinely more robust, or (ii) the generic skill simply has more rules and more relevant trace evidence, including the target's own preferences. Section 4.4 shows D has 25 rules vs 14.15 for B, so the content-size confound is real and not controlled. The abstract's claim that 'broadly transferable procedural knowledge can be more robust than developer-specific preference signals' is the paper's central positive conclusion, yet no condition tests a generic skill that excludes the target developer. The negative result (personalized skill not better than no skill) is not affected by this leak, but the headline generic-skill advantage and the comparison 'personalized vs shared knowledge' are directly undermined.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether personalized skills distilled from a developer's past interaction sessions help a coding agent on the developer's future tasks. It proposes a two-stage pipeline: rule-based bootstrap generation followed by LLM-based, evidence-grounded refinement, producing a SKILL.md document. Evaluation is done by replaying held-out sessions (42 test sessions from 13 developers, five random splits) with a trajectory-conditioned LLM developer simulator. Four conditions are compared: no skill, the target developer's personalized skill, another developer's skill, and a generic skill pooled across all developers. The central results are that the personalized skill improves mean task-completion score by +0.97 (p=.399, win rate 41.43%) over no skill, whereas the generic skill improves by +3.78 (p=.063, win rate 50.95%). The authors conclude that personalized skills provide limited, inconsistent gains while generic skills are more robust, and they identify a precondition for personalization: multiple relevant historical sessions.","tokens_in":15367,"tokens_out":5505,"duration_ms":48944,"significance":"The paper addresses a timely and practical question about LLM coding agents: whether developer-specific personalization extracted from interaction histories transfers to future tasks. The negative result for personalized skills is honestly reported and is consistent with the data. The replay framework is a useful methodological contribution, and the check that simulator follow-ups are semantically consistent with real developer messages (89.47% exact or partial match, Section 4.2) is a good-faith validation step. However, the paper's positive headline conclusion about generic skills is undermined by a design confound: the generic condition is built from a pool that includes the target developer's own evolution sessions, so it is a strict superset of the personalized skill. In addition, the generic-skill advantage is only marginally significant and barely exceeds a chance win rate. The significance of the empirical contribution therefore hinges on whether these issues can be resolved.","major_comments":[{"comment":"The generic skill used in Condition D is explicitly generated from the pooled evolution sessions of all developers, including the target developer. This makes D a strict superset of the personalized skill B, which uses only the target developer's sessions. The comparison in Table 1 (D: 68.80 vs. B: 65.99) therefore cannot distinguish the claim that shared procedural knowledge is more robust from the trivial alternative that D simply contains more rules and more of the target's own trace evidence. Table 4 confirms the content-size confound: D has 25 rules vs. 14.15 for B. The abstract's claim that 'broadly transferable procedural knowledge can be more robust than developer-specific preference signals' is not supported by any condition that excludes the target developer from the pooled skill. A leave-one-out generic condition, built from all developers except the target, is needed to support the intended conclusion.","section":"Section 3.2, Condition D"},{"comment":"The generic-skill improvement over the no-skill baseline has p=.063 and a win rate of 50.95%, which is not statistically significant at the conventional 0.05 level and is barely above the chance level. The abstract states without qualification that generic skills 'achieve the largest and most consistent gains,' while the body properly acknowledges the p-value. Moreover, three pairwise comparisons (A vs. B, A vs. C, A vs. D) are made without multiple-testing correction, so the evidence for the generic-skill advantage is even weaker than the nominal p-value suggests. The central positive claim of the paper should be reframed as a suggestive but non-significant trend, or supported by a pre-registered, corrected analysis with confidence intervals.","section":"Section 3.3, Table 1"},{"comment":"The trajectory-conditioned developer simulator is the core of the evaluation, but Section 4.2 validates only the semantic consistency of the simulator's follow-up messages with real developer messages (exact or partial match in 89.47% of cases). It does not validate that the simulator's acceptance/rejection decisions—i.e., whether a requirement is deemed addressed or still unresolved—match the real developer's decisions in the original session. If the simulator is systematically more lenient under Condition D or more demanding under Condition B, the measured score differences would be artifacts of the simulator rather than effects of the skills. Because the reported effect sizes are small, the paper should validate the simulator's final acceptance decisions against real session outcomes or provide a sensitivity analysis showing that decision consistency does not vary across conditions.","section":"Section 2.3 / Section 4.2"},{"comment":"The subgroup analysis supporting the claim that personalized skills become effective when relevant history is abundant is based on '42 held-out tasks in one random seed' and uses hand-defined bins (0, 1-2, 3-5, ≥6 relevant sessions). The bin with at least six relevant sessions contains only 12 tasks, and no significance testing is reported for the B−D difference of +5.67 or the B−A difference of +10.17. This post-hoc, single-seed analysis is too weak to support the abstract's statement that 'personalized skills become more effective when developer preferences appear frequently.' The analysis should be extended to all five seeds with a correction for multiple testing, or the conclusion should be tempered to a hypothesis-generating observation.","section":"Section 4.1, Table 2"}],"minor_comments":[{"comment":"The abstract text is duplicated in Section 1 with only minor wording differences; remove the duplicate to avoid redundancy.","section":"Abstract and Section 1"},{"comment":"The table formatting is garbled in the manuscript: the header 'NB−A B−C B−D' and row entries like '3−6.33 +15.00−8.00' are ambiguous. Use a standard table with clear column separators and explicit signs for each difference.","section":"Table 2"},{"comment":"The analysis uses only one random seed for the relevant-session binning, while the main results use five seeds; please justify this choice or report across all seeds.","section":"Section 4.1"},{"comment":"The first sentence of the last paragraph reads 'Similarity, comparing...' and should read 'Similarly, comparing...'.","section":"Appendix B"},{"comment":"The sentence 'We use Codex 1 with GPT-5.5 for personalized skill generation...' is unclear about whether 'Codex 1' is the API, the CLI, or a model version; specify the exact model and interface identifiers.","section":"Section 2.3"},{"comment":"The citation for ToM-SWE appears in the text as '[Zhou et al.]' without a year or a corresponding reference entry format match; the reference list entry should be aligned with the in-text citation style.","section":"References"},{"comment":"There is a typo: 'in stead of' should be 'instead of'.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's most interesting result is the null or weak effect of personalized skills, which appears to survive the design issues. The generic-skill advantage, however, is compromised by the inclusion of the target developer in the pooled condition, and a leave-one-out analysis could plausibly weaken or reverse the headline claim. The simulator-decision validity issue is also important because the evaluation uses simulated acceptance as the outcome. I would not reject the manuscript, but the authors must address these confounds before the central claims can be taken at face value. The per-developer subgroup analysis should also be treated as exploratory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central negative claim holds up: personalized skills distilled from a developer's interaction history give small, inconsistent gains over no skill, and the paired numbers (B vs A: +0.97, p=.399, win rate 41%) are reported honestly. That is a useful measurement, and the replay framework is a real step forward for reproducible evaluation of skills in coding-agent sessions. The two-stage skill generation (rule-based bootstrap + evidence-grounded refinement) is sensible, and the paper is careful to split evolution and held-out sessions, reconstruct repository states, and use multiple seeds. Credit where due: this is a genuine empirical contribution on a question prior work mostly left open.\n\nThe soft spot is the headline claim that generic skills are 'more robust than developer-specific preference signals.' As the stress-test notes, Condition D is built from the pooled evolution sessions of all developers, including the target developer, so the generic skill contains all the information in the personalized skill plus more, and Table 4 shows it is substantially larger (25 rules vs 14.15). The D-vs-B comparison does not isolate 'shared knowledge' from 'simply more data and more rules.' This is a load-bearing confound for the paper's positive conclusion, not a minor quibble. The generic advantage also falls short of conventional significance (p=.063), which the paper does disclose but then downplays by leaning on the win rate. That overreach should be corrected, either with a generic skill that excludes the target developer or with much more careful framing.\n\nTwo more concerns. First, the simulator validation only checks semantic consistency of follow-up messages (89% exact/partial match), not whether the simulated developer's acceptance or rejection matches the real developer's. If the simulator is systematically more lenient or more demanding under certain skill conditions, the score differences become an artifact. Second, the subgroup analysis on relevant sessions is post hoc and based on one random seed, so the 'personalization works with enough relevant history' trend is speculative.\n\nThe paper is worth serious peer review, not a desk rejection. The negative result is solid enough to publish and will guide where people put engineering effort. But the authors need to fix the generic-condition confound, temper the abstract, and ideally release code. For citation, I would cite the negative result, not the generic-skills conclusion. I would bring it to reading group for the method and the confound discussion.","headline":"The negative result on personalized skills is credible; the headline 'generic beats personalized' is confounded because the generic skill is built on the target developer's own data, making it a superset rather than a clean comparison.","tokens_in":664,"tokens_out":687,"would_cite":true,"duration_ms":24801,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Personalized coding-agent skills give small, inconsistent gains; pooled generic skills give the largest, most consistent improvement.","keywords":["personalized skills","coding agents","developer interaction histories","agent skills","LLM-based developer simulator","session replay","software engineering agents","empirical evaluation"],"falsifier":"Replace the LLM developer simulator with a deterministic judge that accepts or rejects each candidate response exactly as the recorded real developer did; if the generic skill's +3.78 advantage over no skill fails to replicate under this oracle, the reported advantage is an artifact of simulator leniency rather than a genuine effect of pooled skills.","tokens_in":14919,"feed_emoji":"🤖","tokens_out":4434,"duration_ms":38857,"temperature":0.7,"pith_summary":"The paper asks whether coding agents can be improved by distilling a developer's recurring preferences from past interaction logs into 'personalized skills,' and whether such skills generalize to future tasks. To answer, the authors generate skills with a rule-based bootstrap plus evidence-grounded refinement, then replay 206 real developer-agent sessions using an LLM-based developer simulator. They find personalized skills give small, inconsistent gains over no skill (+0.97, p=.399), while a generic skill pooled across all developers gives the largest and most consistent gain (+3.78, p=.063). The authors conclude that with sparse per-developer histories, broadly transferable procedural knowledge is more robust than developer-specific preference signals; personalization helps only when many relevant past sessions exist.","feed_headline":"Shared skills beat personalized skills for coding agents","feed_subtitle":"Replaying 206 real sessions, pooled skills gained +3.78 vs +0.97 and a 50.95% win rate.","key_machinery":"The central mechanism is a two-stage skill-generation pipeline paired with a trajectory-conditioned replay evaluation. First, a rule-based template produces a bootstrap SKILL.md that captures recurring communication, work-style, follow-up, validation, and commit preferences while excluding task-specific details. Second, an evidence-grounded LLM refiner verifies each candidate rule against the developer's evolution sessions, retaining, revising, or removing rules only when supported by at least two user turns across different sessions. Evaluation then replays each held-out session under identical initial conditions using an LLM-based developer simulator that reads the original task summary and current trajectory and issues focused follow-ups or no further requests. This replay framework isolates the effect of the injected skill by keeping repository state, initial request, simulator, and agent configuration fixed across conditions.","core_discovery":"The paper claims that developer-specific skills distilled from a single developer's interaction history do not reliably improve coding-agent performance on held-out tasks. In their replay experiment, the personalized-skill condition improves mean task-completion score by only +0.97 over the no-skill baseline, with a win rate of 41.43% and p=.399, while a skill from a random other developer gives +0.92 (p=.451). By contrast, a generic skill built from pooled histories of all developers yields the largest gain, +3.78, a 50.95% win rate, and p=.063. The paper further shows that personalized skills become effective only when the developer's history contains at least six sessions relevant to the held-out task, where the gain reaches +10.17 over no skill; below that threshold, generic skills outperform personalized ones. The authors interpret these results as evidence that limited per-developer histories make personalized preference signals unreliable, whereas pooled procedural knowledge transfers more robustly.","pith_inferences":["A natural extension the paper leaves implicit: the right architecture may be a shared skill library with task-relevant retrieval plus a thin developer-specific overlay that only activates when sufficient evidence exists; this follows from the finding that pooled guidance is robust while personalization works only with enough relevant examples.","The reported threshold of at least six relevant sessions is testable: one could prospectively track developers and measure when personalized skills cross zero advantage, which would guide when to spend effort distilling individual preferences.","Because the simulator matches real developer messages semantically in about 89% of cases but the paper does not show that semantically matched messages produce identical agent behavior, a direct human-in-the-loop replication is the natural next check; this is my inference, not a paper claim.","The generic skill's advantage may partly reflect content coverage (25 rules vs. 14 on average for personalized skills) rather than cross-developer transfer per se; the paper's own content analysis suggests this, and a controlled test holding rule count equal would isolate the cause."],"forward_implications":["Skill conditioning can improve coding-agent task completion, but the reliable gains come from pooled, broadly applicable guidance rather than per-developer personalization.","With sparse per-developer histories (fewer than six relevant sessions), personalized skills perform no better than a skill from a random other developer; personalization only pays off when abundant relevant history exists.","Injecting any skill increases compute and interaction: the paper reports more tool calls, tokens, changed files, and patch churn under all skill conditions, alongside more test commands and more successful validation runs.","Evidence-grounded LLM refinement of the bootstrap skill yields only a marginal, non-significant improvement (+0.28, p=.792), so the refinement step is not the driver of the generic-skill advantage."],"supporting_citations":[{"why":"Supplies the 206 real developer-agent sessions and the 100-point scoring rubric used as the dataset and outcome measure.","marker":"[Baumann et al., 2026]"},{"why":"Provides the interactive replay design that the paper adapts for trajectory-conditioned developer simulation.","marker":"[Wu et al., 2026]"},{"why":"Supplies the trace-to-skill distillation method that the evidence-grounded refinement stage builds on.","marker":"[Ni et al., 2026]"},{"why":"Motivates iterative, evidence-grounded refinement of natural-language skill documents.","marker":"[Yang et al., 2026a]"},{"why":"Background for improving skills via trajectory-level diagnosis, cited in the refinement design.","marker":"[Wang et al., 2026]"},{"why":"Prior work distilling recurring preferences from dialogue histories into reusable skills, the personalization approach this study contrasts with.","marker":"[Yang et al., 2026b]"}],"fun_headline_variants":["Personalized coding skills don't reliably beat no-skill baseline","Shared skills outperform personalized for coding agents","Pooled skills beat personal ones in coding agent study","Why generic skills win over personalized for coding agents","Coding agents gain more from shared than personal skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the simulated developer judging agent responses the way the real developer would; if the simulation is more lenient for generic-skill runs, the measured advantage would be an artifact of the simulator rather than of the skill content.","fun_headline_variants_meta":{"raw":{"variants":["Personalized coding skills don't reliably beat no-skill baseline","Shared skills outperform personalized for coding agents","Pooled skills beat personal ones in coding agent study","Why generic skills win over personalized for coding agents","Coding agents gain more from shared than personal skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1688,"prompt_tokens":1013,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":601}},"tokens_in":629,"tokens_out":675,"duration_ms":6932,"temperature":1.0,"reasoning_tokens":601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:36.821404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the LLM developer simulator with a deterministic judge that accepts or rejects each candidate response exactly as the recorded real developer did; if the generic skill's +3.78 advantage over no skill fails to replicate under this oracle, the reported advantage is an artifact of simulator leniency rather than a genuine effect of pooled skills.","supporting_citations":[],"review_version":1}