Machine Learning · arXiv:2608.12307 v1 · submitted 12 Aug 2026 · 15 references
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
The paper says
“Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods.”
more of the abstract
In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A real, well-measured phenomenon with an overstated headline: the missing decoding-matched baseline makes the top-line uplift uninterpretable as scaffold-specific, but the direction and the mechanistic story deserve referee time. This line is wrong
5 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 15 references say.
The paper says, at §4 Baselines; §5.4 Fig. 6(a)
The review did not quote this passage.
Pith says: Objection 1 of 5 · major
The vanilla baseline's decoding parameters are not reported anywhere in §4, while 56 of 57 scaffolded runs use greedy decoding at temperature 0 (§5.4, Figure 6(a)). If the baseline call used stochastic sampling, the headline uplift (mean +0.275; 0.488→0.912 best run) conflates scaffold content with the decoding change; the paper itself classifies 'Greedy / temp control' as a scaffold technique, and the 56/1 contrast in Figure 9 is too unbalanced to identify its contribution.
more of objection 1
The authors should report the baseline decoding settings and add a greedy-decoding (and ideally format-parsed) vanilla baseline so that the harness-specific contribution is cleanly identified. The McNemar fix/break asymmetry and the per-benchmark gains would likely survive such a control, but the magnitude attribution in Table 1 is not yet identified.
The paper says, at Abstract; §5.1 Table 1
The review did not quote this passage.
Pith says: Objection 2 of 5 · major
The abstract states that test-time transfer 'nearly double[s] average target-model performance from 0.49 to 0.91,' but Table 1 reports the mean over all scaffolded runs as 0.763 (+0.275, a 56% relative increase); 0.912 is the best single run rather than the average. The abstract's headline number therefore overstates the central result and should be revised to match Table 1, e.g., 'from 0.49 to 0.91 for the best run, with a mean of 0.76 across all runs.'
The paper says, at §5.6 Fig. 8(b)
The review did not quote this passage.
Pith says: Objection 3 of 5 · major
The 'headroom law' (Pearson r = 0.75 between realized uplift and headroom 1 − baseline) is partially mechanical because both quantities are defined from the same baseline term: uplift = scaffolded − baseline, while headroom = 1 − baseline, so any scaffold that yields roughly constant absolute accuracy across benchmarks generates a strong correlation by construction.
more of objection 3
Before claiming that scaffolding 'recovers latent competence,' the authors should control for this mechanical link (e.g., regress scaffolded accuracy on baseline and test whether the slope is significantly below 1, or compute the correlation within each target model), and should state what correlation a null model with constant scaffolded accuracy would produce.
The paper says, at §5.9 Fig. 11(a), Table 4
The review did not quote this passage.
Pith says: Objection 4 of 5 · major
The claim that deterministic offloading is 'the mechanistic core of strong-to-weak scaffolding' rests on the determinism-fraction/accuracy correlation (r = 0.72), but this statistic is computed on only 18 runs (Table 4) out of 57 GPT-5.4-mini runs, and the paper does not state how these provenance-tagged runs were selected or whether the 18 are representative.
more of objection 4
In addition, builder strength is a plausible confound: stronger builders may both write more deterministic code and construct better prompts. The authors should report the subset-selection rule and show that the correlation survives within builder, or with builder identity as a covariate, before presenting offloading as the primary mechanism.
The paper says, at §3–§4
The review did not quote this passage.
Pith says: Objection 5 of 5 · major
The test set is described as 'hidden,' but the four benchmarks are public datasets that are likely present in the pretraining corpora of the builder models, and the 5% validation slice is itself drawn from the same public benchmarks; a builder that partially memorized these datasets could design scaffolds that exploit benchmark-specific memorized regularities rather than transferable structure.
more of objection 5
The internal checks (small validation–full gap, r = 0.96) address overfitting to the validation slice but not memorization of the benchmark family. The authors should discuss contamination risk and, ideally, evaluate scaffolds on a fresh sample generated after the builder's knowledge cutoff or on rephrased instances to bound this effect.
5 minor objections
The paper says, at Appendix A
Pith says: minor objection 1 of 5
The builder instruction file in Appendix A tells the builder that the validation set is 'a random 2% sample of the full benchmark,' while §3 and §4 state that the validation split contains 5% (195 items); this factual inconsistency in a method-critical artifact should be reconciled.
The paper says, at §5.8, Fig. 10(a)
Pith says: minor objection 2 of 5
The text reports 'χ² ≫ 10^4' for the McNemar test, but the values shown in Figure 10(a) are 1424.4 and 600.3; the text should read χ² ≈ 1.4×10^3 (and p < 10^-4).
show 3 more
The paper says, at Table 6
Pith says: minor objection 3 of 5
In Table 6 the 'Avg.' and 'Val.' columns are concatenated without a separator (e.g., '0.91825' and '0.83715'), which makes the run-level data needlessly hard to parse; please add a delimiter or use separate columns.
The paper says, at Fig. 3(b)
Pith says: minor objection 4 of 5
The per-benchmark reference values printed in Figure 3(b) do not obviously match the baseline row of Figure 3(a) (e.g., Hi-ToM baseline 0.569, MMToM 0.412, MuMA 0.469 versus the 0.64/0.70/0.62/0.75 labels); please verify that the bars and labels correspond to the correct reference points.
The paper says, at §1, §2, and throughout
Pith says: minor objection 5 of 5
Minor language issues: 'we investigate into this strong-to-weak scaffolding' (§1) and 'Our work does not intend as a new ToM benchmark' (§2) should be rewritten, and 'monotone' should be 'monotonic' where used. The provided text also contains numerous missing-space artifacts (e.g., 'TheVanillabaseline,' 'study strong-to-weak'), so the authors should ensure the published version is clean.
What would settle it
Rerun the GPT-5.4-mini vanilla baseline inside the same evaluation harness used for scaffolded runs, holding prompt, API, max tokens, and temperature 0 fixed, with at least 3 independent seeds; symmetrically, execute one representative scaffold (e.g., GPT-5.5/GPT Codex) at the baseline temperature. If the greedy vanilla baseline remains near 0.488 and the scaffold at baseline temperature stays near 0.763, the decoding confound is dismissed. If the greedy baseline rises materially above 0.488, or the scaffold-at-baseline-temperature result drops below 0.763, recompute the reported uplift and re-estimate the scaffold-specific contribution.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Aug 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 5 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Read the AI4AI paper. The short version: there is a genuine and interesting phenomenon here, and the paper's own controls mostly support its direction; but the single most important number in the abstract—the near-doubling from 0.49 to 0.91—is not cleanly identified, and a rerun under matched decoding is needed before I'd trust the magnitude.
What is new: the paper defines a clean setting (strong builder, fixed weak target, 5% validation, hidden test) and measures, over 72 runs, how much of an inference-time harness is transferable. The technique taxonomy (format enforcement, routing, deterministic solvers, polarity logic) and the determinism-fraction analysis (r=0.72 with accuracy, n=18) are genuinely informative. The builder-vs-platform decomposition, the monotone effort curve for Opus, and the target/headroom analysis are useful measurements even if some are associational. The per-run table in the appendix is a real asset; you can re-analyze their data from it.
Soft spots, in order:
1. The vanilla baseline does not appear to control decoding. Section 5.4 says 56 of 57 scaffolded runs use temperature 0; Section 4 describes the baseline only as a "direct call with a naive prompt." If the baseline used stochastic sampling, then the +0.275 mean and the 0.488→0.912 best-run claim conflate scaffold content with variance reduction. The paper itself lists greedy/temp as a scaffold technique, so this is not a nitpick; it is the central magnitude. The McNemar asymmetry (1717 fixed vs 105 broken) suggests the effect is not all decoding, but the size is not identifiable without a matched baseline.
2. The "headroom law" (r=0.75 between uplift and 1−baseline) is partly mechanical, because baseline appears on both sides of the correlation. I'd like to see a version that conditions on scaffolded accuracy or reports per-benchmark partial residuals.
3. No artifacts or splits released; no contamination check on the public ToM benchmarks. For a paper this dependent on hidden-test integrity, that is a real gap.
Minor: the body says 5% validation, Appendix A says 2%. Also the abstract's "nearly doubling" refers to the best run, not the mean; the mean uplift is +0.275 (a 56% relative gain), which is still large but not "nearly doubling."
The central claim—that a strong model can build reusable scaffolds that lift a weak target on held-out ToM data—holds up in the sense that 100% of runs beat baseline and the per-benchmark improvements are coherent. The paper is transparent about its associational attributions and residual-error analysis.
Who this is for: anyone working on scaffold/harness optimization, weak-to-strong transfer, or evaluation methodology. It deserves a serious referee, with the decoding-matching issue made a required revision.
Would bring it to reading group, yes.
The strongest objection
The most load-bearing assumption is that the vanilla baseline is a fair counterfactual. Section 4 specifies the baseline only as a direct call with a naive prompt and reports 0.488 for GPT-5.4-mini, while Section 5.4 and Figure 6a show that 56 of 57 scaffolded runs use greedy/temperature-0 decoding. If the baseline used stochastic sampling, the reported mean uplift (+0.275), the 100%-of-runs claim, and the best-run nearly doubling (0.488 to 0.912) all mix two separable effects: scaffold content (routing, format enforcement, deterministic offloading) and decoding variance reduction. The paper's own attribution analysis in Section 5.8 lists 'Greedy / temp control' as a scaffold technique, making the missing matched control a genuine confound rather than a minor reporting gap. This does not make the central direction false; the McNemar fix/break asymmetry (1717 fixed vs. 105 broken) and the per-benchmark monotonic effects would likely survive. But the headline magnitude is not cleanly identified until the baseline is rerun under identical decoding.
The reader's weakest assumption is precisely the one I consider most load-bearing: the paper's own technique table shows temperature control is near-universal in scaffolds but absent from the reported baseline description. This is the kind of condition that, if wrong, directly changes the paper's headline magnitude rather than merely its interpretation. I do not see a more serious internal flaw: the held-out protocol is described clearly, the validation-to-full optimism gap is small, the McNemar tests are appropriate for paired item-level changes, and the technique-attribution analysis is explicitly framed as associational. The decoding mismatch is easily addressable with a matched-baseline rerun, so the current CONDITIONAL verdict remains appropriate; no adjustment is needed.
The referee's report
If the results hold, this is a substantial empirical contribution: it demonstrates a reproducible test-time capability-transfer axis and ships complete run-level data (Table 6) plus several checks that strengthen the core measurement — an externally evaluated hidden set, item-level McNemar tests with a strongly asymmetric fix/break ratio (1717 vs 105), a small validation–full optimism gap (0.021), a self-scaffolding control showing +0.17–0.22 even for a weak builder, and a documented failure mode in which scaffolding hurts a near-ceiling target. The main risks concern effect-size attribution rather than the direction of the effect: the vanilla baseline's decoding settings are unreported while 98% of scaffolds use greedy decoding, the headroom 'law' has a mechanical component, and the determinism-mediated mechanism rests on a non-random subset of runs. These issues are fixable with additional controls, and they do not appear to overturn the central direction of the result.
Pith's reading of the paper (a machine's summary, not the authors')
The paper asks whether a strong model can transfer capability to a weaker one without retraining it, by building an inference-time harness that the weak model executes. On four Theory-of-Mind benchmarks, the authors report that scaffolding raises a weak target's macro-average accuracy from 0.488 to an average of 0.763, with the best run reaching 0.912, and that all 57 scaffolded runs beat the un-scaffolded baseline. The builder sees only a 5% validation slice and never the hidden test set, so any gain must come from reusable task structure rather than memorized answers. The paper argues that the main mechanism is cognitive-load reduction: offloading unstable reasoning into deterministic code, routing, and strict answer-format enforcement, rather than inducing longer reasoning or broader sampling. If correct, this makes harness design a practical complement to distillation and a new way to measure how well a strong model can externalize its competence.
What carries the argument. The load-bearing object is the scaffold (harness): an executable inference-time structure, written in code, that routes tasks, enforces answer formats, and sometimes solves subproblems deterministically. The builder iterates over it using the validation slice as a proxy: it inspects the validation set, proposes or revises the scaffold, evaluates it on the validation slice, diagnoses errors, and repeats until submission, a loop summarized in the paper's Algorithm 1. The identified mechanism inside the scaffold is cognitive-load reduction, operationalized as the determinism fraction—the share of the 3,900 evaluation items answered entirely by code or structured rules—which correlates with final accuracy at r=0.72. The other central relation is the headroom law: realized uplift tracks 1 minus baseline accuracy (r=0.75), which explains why weaker targets gain most and why over-scaffolding a near-ceiling target can regress it.
Core claim. The paper's central claim is that strong-to-weak scaffolding—a stronger builder model writing an inference-time scaffold around a fixed weaker target model—works as a form of test-time capability transfer. Holding the target at GPT-5.4-mini, the mean macro-average accuracy across all builder-platform-repeat runs is 0.763, an uplift of +0.275 over the 0.488 vanilla baseline, with 100% of runs and all 11 builder configurations exceeding the baseline and the best run (GPT-5.5 on GPT Codex) at 0.912. The authors find the gains are driven chiefly by deterministic offloading, benchmark-aware routing, and format enforcement (determinism fraction correlates with accuracy at r=0.72), and that builder reasoning effort improves scaffold quality monotonically while platform effects are second-order. Weaker targets gain more than stronger ones, with realized uplift tracking the target's headroom (r=0.75), and residual errors concentrate in deep belief recursion and Bayesian goal inference—the least compilable parts of the tasks.
The premise it rests on. The uplift numbers assume the vanilla no-scaffold baseline is a matched counterfactual: if the baseline used stochastic sampling while the scaffold runs forced temperature-0 greedy decoding, part of the measured gain would come from the decoding change rather than from the scaffold itself.
If the paper is right (Pith's inferences, not the authors')
Scaffolding a weak model at inference time can outperform switching to a much larger unscaffolded model: the best scaffolded GPT-5.4-mini run beats raw GPT-5.4 and GPT-OSS-120B on all four benchmarks.
Because validation accuracy tracks full-set accuracy almost one-to-one (r=0.96) with a small optimism gap, a 5% validation slice is enough to guide harness construction without substantial overfitting.
Builder capability and reasoning effort, not the agentic coding platform, are the main levers: for Opus-4.7, raising reasoning effort from low to extra-high increases macro accuracy monotonically from 0.711 to 0.856.
The recipe transfers unevenly: weak targets benefit everywhere, while already-strong targets gain almost only on the most compilable benchmark and can regress on near-ceiling sub-tasks, so scaffolding should be gated by measured headroom.
Deterministic offloading is the best predictor of scaffold quality, implying that harness design should prioritize converting task structure into code over eliciting more reasoning from the target.
A natural extension is a builder-evaluation benchmark: score a builder not on its own answers but on how much it improves a fixed weak target under a fixed validation budget; this would measure the ability to externalize reasoning, a capability direct task benchmarks may miss.
The headroom law suggests a concrete deployment rule: before scaffolding a sub-task, estimate the target's headroom on it, and apply scaffolds selectively where headroom is positive; near-ceiling sub-tasks should be passed through untouched.
Because the determinism fraction is measured from provenance tags, a direct test of the cognitive-load story would log actual target-model inference steps or API calls per item and show that scaffolded runs displace model computation onto code.
The monotone builder-effort result implies diminishing but positive returns to builder deliberation; a cost-based extension would trade builder reasoning tokens against target-model savings to find the economically optimal effort tier.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.93
“These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.”
How the main claim is stated: hedged possibility 0.99
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.32 · human participants 0.03 · controlled comparison 0.74 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.84 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. | background 1.00 | 0.03 | 0.02 | asserted 0.99 |
| In this paper, we ask whether such transfer can instead occur at test time. | gap 0.75 | 0.16 | 0.03 | hedged possibility 0.40 |
| We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. | contribution 0.98 | 0.30 | 0.09 | hedged possibility 0.61 |
| Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. | method 1.00 | 0.95 | 0.03 | asserted 0.93 |
| Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. | result 1.00 | 0.97 | 0.40 | asserted 1.00 |
| Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. | interpretation 0.87 | 0.96 | 0.26 | asserted 1.00 |
| We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. | result 0.99 | 0.97 | 0.08 | asserted 1.00 |
| These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining. | interpretation 1.00 | 0.56 | 0.87 | hedged possibility 0.99 |
What kind of paper the classifier reports this is: method or tool 0.64 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,407 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
References (15)
- [1] **Average performance across the four benchmarks** -- primary criterion on Pith
- [2] **Number of validation evaluations used** -- secondary criterion. Each validation evaluation costs 1. Use validation runs carefully. Do not repeatedly test small or unprincipled changes. Your scaffold should improve the target model's task performance and reduce its cognitive ... on Pith
- [3] Inspect the workspace, especially`validation.jsonl`and`engines.py` on Pith
- [4] Understand the input/output format, benchmark types, and evaluation requirements on Pith
- [5] Build an initial scaffold and baseline evaluation script on Pith
- [6] Run validation only after meaningful scaffold changes on Pith
- [7] Analyze failures and refine the scaffold recursively on Pith
- [8] Stop when performance plateaus or further changes risk overfitting on Pith
- [9] Prepare the final full-set evaluation script. Your scaffold may include prompt templates, task routing, answer-format enforcement, few-shot examples, verification steps, deterministic preprocessing/postprocessing, or other methods that help the target model perform better. Avo... on Pith
- [10] Load the given full-set file on Pith
- [11] Run your final scaffold on it on Pith
- [12] Save predictions/results inside the workspace on Pith
- [13] Append the full-set benchmark results to`performance.csv` on Pith
- [14] Full set testing should be parallel call of target models with max worker equals 16 on Pith
- [15] Full set final results or analysis output should be saved in independent folder`final_eval` ## Final Reminder You have only one final chance on the hidden full set. Be strategic: inspect carefully, make each validation run count, recursively improve the scaffold, and prioritiz... on PitharXiv
Cited by 6 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Teaching the Harness: Student-Aware Skill Transfer for LLM Agents cs.AI · Nov 2026
- Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep? cs.AI · Oct 2026
- Scaling Trajectories for Complex Tasks through Recursive Self-Rewrite cs.AI · Oct 2026
- Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI cs.AI · Sep 2026
- From Migration to Calibration: Aligning Agents with Deployment Requirements cs.AI · Sep 2026
- Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search cs.AI · Aug 2026
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Cites it as background 1 paper
-
AI4AI [ref] studied how a teacher model builds an inference-time harness for weaker student models.
Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search · Introduction · 0.99
What it cites · 10 sentences without a confident label
Uses the method or tool of 1 paper
-
This setting yields a macro-average accuracy of $$ for GPT-5.4-mini and $$ for Gemini-3.5-flash. * Human-Inspired Harness: Each target model is evaluated with UserHarness [ref] , a human-designed harness framework for ToM problems.
UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind · Experimental Setup · 0.86
Cites as background 6 papers
-
Chain-of-thought prompting elicits intermediate reasoning steps from large models [ref] , self-consistency improves reliability by sampling multiple reasoning paths and marginalizing over their answers [ref] , and least-to-most prompting decomposes hard problems into easier subproblems solved sequentially [ref] .
Self-Consistency Improves Chain of Thought Reasoning in Language Models · Related Work · 0.96
-
Chain-of-thought prompting elicits intermediate reasoning steps from large models [ref] , self-consistency improves reliability by sampling multiple reasoning paths and marginalizing over their answers [ref] , and least-to-most prompting decomposes hard problems into easier subproblems solved sequentially [ref] .
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models · Related Work · 0.96
-
Recent progress in model distillation has made it increasingly plausible to deploy smaller language model experts in settings that once required much larger ones [ref] .
Distilling the Knowledge in a Neural Network · Introduction · 0.87
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment