{"paper":{"title":"Harness Evolution Hits a Ceiling: When Weight Training Should Begin","arxiv_id":"2610.11655","primary_cat":"cs.AI","submitted_at":"2026-10-08T10:33:29+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":0.99,"id":"s0","own":false,"role":"background","text":"Improving a long-horizon LLM agent means evolving the harness around a frozen model or training its weights.","hedge":{"p":0.96,"label":"asserted"},"own_p":0.1,"main_claim_p":0.05,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.99,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":1,"p":0.87,"id":"s1","own":true,"role":"method","text":"We let a self-evolving harness make the system stronger first, then cross seed and evolved harnesses with base and trained weights to learn which gains the trained model keeps and which still need the runtime.","hedge":{"p":0.8,"label":"asserted"},"own_p":0.92,"main_claim_p":0.08,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.0,"method":0.87,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.13,"interpretation":0.0}},{"i":2,"p":0.85,"id":"s2","own":true,"role":"method","text":"We show that the right lever can be read off the agent's failure composition: labelling failed trajectories by the first signal that fires separates process failures (blocked calls, loops, exhausted step budgets) from content failures (a delivered plan that is poor).","hedge":{"p":1.0,"label":"asserted"},"own_p":0.9,"main_claim_p":0.12,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.0,"method":0.85,"result":0.01,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.14,"interpretation":0.0}},{"i":3,"p":0.92,"id":"s3","own":true,"role":"interpretation","text":"Harness evolution repairs the former, the behaviour it instils can be trained into the weights, and content failures are what weight training is for.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.69,"main_claim_p":0.64,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.0,"method":0.05,"result":0.01,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.02,"interpretation":0.92}},{"i":4,"p":1.0,"id":"s4","own":true,"role":"result","text":"On DeepPlanning, a self-evolving harness loop lifts the held-out score of Qwen3.5-4B from 0.16 to 0.30 and of Qwen3.5-9B from 0.32 to 0.44; for 4B, held-out delivery rises from 55% to 90% while content failures are left for the weights.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.13,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":5,"p":1.0,"id":"s5","own":true,"role":"result","text":"LoRA adapters trained on evolved-harness trajectories internalise the gain: under the original harness they add +0.13 on held-out tasks for both sizes; on 4B they stack with the harness to more than double the held-out score, and on 9B the adapter alone matches the full evolution line, cutting content failures from a quarter of trajectories to one in twenty.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.16,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":6,"p":1.0,"id":"s6","own":true,"role":"result","text":"A placebo adapter trained on answer-shuffled trajectories falls below the base model.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.03,"quantitative":true,"quantitative_p":0.92,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":7,"p":0.98,"id":"s7","own":true,"role":"result","text":"The loop transfers to WebArena-Lite (+0.09 on 117 unseen tasks), where the gain lives in what the model sees and adapters do not add to it.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.07,"quantitative":true,"quantitative_p":0.98,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.98,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.02}},{"i":8,"p":0.52,"id":"s8","own":true,"role":"method","text":"The result is a diagnose-then-intervene rule applied twice: read the failure composition to choose between harness and weights, then read what the accepted edits changed to decide which gains to train in.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.88,"main_claim_p":0.7,"quantitative":false,"quantitative_p":0.33,"role_probabilities":{"gap":0.0,"method":0.52,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.31,"interpretation":0.17}},{"i":9,"p":1.0,"id":"s9","own":true,"role":"method","text":"Scores are four-rollout means against fresh anchors, same-night except where marked, across eight models from six families and two benchmarks.","hedge":{"p":0.9,"label":"asserted"},"own_p":0.9,"main_claim_p":0.03,"quantitative":true,"quantitative_p":0.93,"role_probabilities":{"gap":0.0,"method":1.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":0.72,"text":"The result is a diagnose-then-intervene rule applied twice: read the failure composition to choose between harness and weights, then read what the accepted edits changed to decide which gains to train in.","location":"abstract s8","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":7957,"cost_usd_reported":0.0},"compiled_at":"2026-10-09","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.04,"states_limitation":0.46,"human_participants":0.02,"controlled_comparison":0.96,"randomized_assignment":0.03,"states_quantitative_result":0.99,"compares_with_prior_results":0.9,"mentions_code_or_data_availability":0.04},"evidential_basis":{"p":0.97,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.97,"simulation_or_computation":0.03}},"contribution_type":{"p":0.59,"value":"method or tool","probabilities":{"review":0.0,"method_or_tool":0.59,"empirical_study":0.41,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":0.0,"position_or_commentary":0.0}},"contribution_statement":{"p":0.89,"text":"We let a self-evolving harness make the system stronger first, then cross seed and evolved harnesses with base and trained weights to learn which gains the trained model keeps and which still need the runtime.","location":"abstract s1"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":10,"sentences_in_abstract":10},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}