Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

Da-Tian Peng; Jingkun Luo

REVIEW 4 major objections 4 minor 13 references

Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-07-31 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A correct-answer score does not say why an agent succeeded: the paper introduces success provenance and audits it by comparing matched CLEAN, GOLD, and SHAM conditions.

desk verdict Careful, honestly scoped controlled experiment that gives agent evaluation a reusable test for whether scores depend on evaluation-time target values; the SHAM control isn't neutral, but the main inference survives. read the letter →

arxiv 2607.24054 v1 pith:KUWAZ7XP submitted 2026-07-27 cs.AI

Jingkun Luo , Da-Tian Peng This is my paper · ORCID

classification cs.AI

keywords successprovenanceagentevaluationvaluesubstitutionCLEAN-GOLD-SHAMbenchmarkauditingretrievalagentsmodelcomparisonAcquaBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that outcome accuracy cannot distinguish intended reasoning from answer acquisition during evaluation. It introduces 'success provenance' as a missing evaluation object and tests it with a matched three-condition intervention: CLEAN keeps authorized information, GOLD adds the correct target value, and SHAM adds a matched incorrect value. Across a multi-hop benchmark and three open-weight models, GOLD exceeds SHAM by 19.1 to 25.9 percentage points in the direct-acquisition regime, showing that success follows the correct value beyond source exposure. Under distributed two-source sufficiency, the behavioral contrast persists (11.8 and 14.6 points) even when a single-source lexical detector fails. If right, benchmarks should report whether the evaluated information state supported success, not just whether the answer was correct.

What carries the argument

The central object is the matched CLEAN/GOLD/SHAM value-substitution design. CLEAN exposes only benchmark-authorized evidence; GOLD adds a controlled source containing the correct target value; SHAM preserves source structure and exposure opportunity while substituting a deterministic matched incorrect value. The key identity is the paired contrast GOLD minus SHAM: a positive value attributes the score response to target correctness beyond matched source exposure. Supporting machinery includes the premature-sufficiency construction (D0), the distributed two-source composition (D2), and the coloc score, a lexical overlap of question tokens and predicted-answer tokens in a single observed sour

What would settle it

Run the same frozen D0 qids with four conditions: CLEAN, GOLD, SHAM, and NEUTRAL (a controlled source of matched form that provides no usable information about the question). If GOLD minus SHAM is large while GOLD minus NEUTRAL is near zero, the SHAM decoy itself is interfering and the claimed target-value attribution fails. If GOLD minus SHAM is approximately equal to GOLD minus NEUTRAL, then SHAM interference is negligible and the attribution stands.

Watch

Extended reading notes

Core claim

The central claim is that success is not self-explanatory: once an agent can change its information state, correctness no longer tells whether the score depended on a target value acquired inside the evaluated trajectory. The paper establishes this by holding the question, model, action channel, and scorer fixed and varying only the supplied target value. In the direct single-source regime, GOLD minus CLEAN ranges from 11.3 to 18.0 percentage points, and the matched correct-versus-incorrect contrast GOLD minus SHAM ranges from 19.1 to 25.9 points, with all model-level intervals excluding zero. Under distributed sufficiency, GOLD still exceeds SHAM by 11.8 and 14.6 points, while the single-so

Load-bearing premise

The matched SHAM condition must preserve everything about the GOLD intervention except the correctness of the value; if the wrong value distracts, confuses, or changes retrieval behavior in a way a neutral no-information control would not, then GOLD minus SHAM conflates benefit from the correct value with interference from the incorrect one.

Editorial extensions

If this is right

If success tracks target correctness in this matched sense, outcome-only leaderboards conflate earned success with acquired success and overstate capability.
Behavioral value dependence can persist even when no single source is sufficient, so single-source exposure detectors can miss real acquisition.
Exposed scoring can compress or reverse point gaps between models without establishing rank inversion, so model comparisons should be audited under different information conditions.
Answer form and scorer sensitivity can change measured effects, marking a measurement boundary that should be reported alongside acquisition results.
Benchmark reports should accompany accuracy with a provenance view: the CLEAN condition and the GOLD-minus-SHAM contrast.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The GOLD-minus-SHAM contrast conflates benefit from the correct value with interference from the incorrect value; a neutral, uninformative source control could bound the interference term and sharpen the attribution.
Editorial inference: If the method generalizes, benchmark designers could adopt a standard 'provenance report' that separates authorized-context performance from acquired-target dependence, changing how agent capabilities are compared.
Editorial inference: The D2 AUROC below 0.5 suggests the coloc score's orientation is regime-dependent; a detector that is agnostic to the information structure or trained on distributed examples might recover visibility where this fixed rule fails.
Editorial inference: The frozen populations are small and drawn from one benchmark family; extending the audit to larger, multi-benchmark sets would test whether the 19-to-26-point margins and the D2 persistence are stable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the quick take: this paper builds a controlled, within-question audit asking whether an agent's score follows the correct target value supplied during evaluation. The D0 result is strong — GOLD beats SHAM by 19.1 to 25.9 points across three models, all 12 per-surface estimates positive, and the qid-clustered bootstrap intervals exclude zero. The design is careful, and the paper is unusually honest about its boundaries.

What's new is the matched CLEAN/GOLD/SHAM value-substitution protocol with frozen populations, four standardized action surfaces, and joint qid-clustered analysis. It addresses a real blind spot: exposure detection tells you a source was seen, but not whether success depended on the correct value. D2 adds a clever boundary — distribute the answer across two sources, and behavioral dependence persists even though the single-source coloc signature inverts, with AUROC below 0.5. The model-comparison result is a clean illustration of how raw GOLD scoring can compress a supported CLEAN gap.

The main weakness is the SHAM control, as the stress-test note says. SHAM accuracy is 6–8 points below CLEAN for every model, so the GOLD-minus-SHAM contrast mixes benefit from the correct value with harm from the decoy. The paper explicitly concedes it doesn't separate those, and I don't think the existence claim — that scores track value correctness — requires that separation. A decoy that interferes is itself evidence of value sensitivity. But if the authors want to claim GOLD-minus-SHAM measures "support" rather than "sensitivity," they need a neutral no-information control or at least a bounded analysis of decoy interference. That's a revision point, not a fatal flaw.

D2 rests on only 36 qids and 12–25 positives, with wide CIs; the authors scope it as a boundary result, and the AUROC inversion is descriptive. D1 is rightly treated as a measurement boundary, not positive evidence. The whole thing is limited to HotpotQA and one orchestration setup, which they say plainly.

This paper deserves a serious referee. The methodology is reusable, the write-up is transparent, and the claim-boundary matrix plus reproduction details go well beyond the norm. I'd send it out; the SHAM-control issue is exactly what a good reviewer should push on, either toward a neutral condition or a sharper wording of what the contrast identifies.

Referee Report

4 major / 4 minor

Summary. The paper defines 'success provenance' as the missing evaluation object in agent benchmarks: whether an observed success depended on a target value acquired during the evaluated trajectory, rather than on benchmark-authorized information. It introduces AcquaBench, a within-qid matched CLEAN/GOLD/SHAM value-substitution audit on HotpotQA, across four standardized action surfaces (retrieval, tool, memory, subagent), with qid-clustered percentile-bootstrap inference on frozen artifact-eligible populations. In D0 (direct single-source sufficiency), GOLD exceeds SHAM by 19.1–25.9 percentage points across Llama-3.1-8B, Qwen2.5-14B, and Qwen2.5-32B, and a style-controlled validation reproduces the pattern. In D2 (distributed two-source sufficiency), GOLD exceeds SHAM by 11.8 and 14.6 points, while the single-source coloc score no longer transfers as a high-score marker (AUROC 0.376 and 0.142). A within-Qwen model comparison shows a supported 5.0-point CLEAN gap compresses to -0.6 points under raw GOLD, without establishing rank inversion. The paper is carefully scoped with an explicit claim-boundary matrix, and it reports code, frozen configurations, and provenance checks.

Significance. If the central contrast is valid, the paper offers a practical audit for agent evaluation that goes beyond outcome accuracy and exposure detection, and it provides evidence that target-value dependence can persist beyond a single-source detector's observation unit. The design's strengths are substantial: prespecified contrasts, frozen populations, qid-clustered bootstrap, style controls, per-surface descriptive transparency, and a machine-checkable release protocol. The main numerical claims are internally consistent and the paper does not overclaim; for example, it explicitly avoids asserting rank inversion or a scaling law. The contribution would be of clear interest to the agent-benchmarking community, and the open repository strengthens reproducibility.

major comments (4)

[AcquaBench Design / Supplementary B.2] The central estimand ΔGS is interpreted as showing that success tracks target correctness beyond matched source exposure. But ΔGS = ΔGC − ΔSC, and in D0 ΔSC is negative for every model (Table 2: SHAM−CLEAN is −8.0, −7.0, −7.7 pp). A negative ΔSC means the decoy harms performance relative to CLEAN, so ΔGS conflates benefit from the correct value with interference from the incorrect value. The manuscript concedes this ('It does not separately identify help from the correct value and interference from the incorrect value') and states in A.1 that SHAM does not assert identical realized attention or action paths. Without a neutral-source condition or an independent bound on the interference term, the headline 'success follows the correct value' is not uniquely supported. This is load-bearing because ΔGS is the paper's main value-dependence statistic.
[Supplementary A.5 / Table 3] The D2 claim that behavioral dependence persists after answer sufficiency exceeds a single-source observation unit rests on the premise that 'max single-source construction score ≤ 0.25' ensures no single source answers the question. The gate is applied to construction artifacts using the supplied target value, whereas the runtime detector uses the model's predicted answer and observed sources (B.5). The paper itself notes these are not the same random variable. With only 36 qids and 12–25 positives, I would like explicit evidence—or a released audit—that no individual Gold-exposed source in the actual observed trajectories was answer-sufficient, and that the two sources must be composed. The two mechanism cases help, but they are only two trajectories.
[Results: D1 / Table 3] The D1 boundary results show a large positive GOLD−SHAM for Llama (+15.1 pp) despite a negative GOLD−CLEAN (−2.0 pp), and a near-zero/negative GOLD−SHAM for Qwen (−2.0 pp) in the same construction. The paper treats D1 as a measurement boundary, which is a reasonable reading, but it also illustrates that the value-substitution effect is not invariant to answer form and scoring. This reinforces the first concern: ΔGS can be driven by surface-level interactions rather than target correctness alone. I would ask the authors to state more explicitly how D1 constrains the generalizability of the D0 'success provenance' interpretation to other scoring conventions.
[Table 4 / Model-comparison audit] The model-comparison claim is appropriately qualified ('does not establish rank inversion'), but the practical reading in the abstract—'a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points'—could be strengthened by reporting the joint distribution of the two gaps rather than two marginal intervals. Since the same qid clusters underlie both quantities, a cluster-bootstrap test of whether the CLEAN gap exceeds the GOLD gap would directly address whether the compression is statistically meaningful. This is a suggestion for additional analysis rather than an objection to the current phrasing.

minor comments (4)

[Table S1] The D2 row in Table S1 appears to list '50 44 36' in the parent/template/paired columns; the main text says a separately frozen 50-question parent sample with 36 paired qids. Please reconcile the '44' entry or clarify what it denotes.
[Abstract / Introduction] The term 'coloc' is defined in the body but not in the abstract; readers of the abstract will not know what AUROC 0.376 and 0.142 refer to. Consider a brief gloss.
[Supplementary E.2] The prompt template shows a user exemplar with '<ACTION>' placeholder; the exact substitution for each action surface (Search, Lookup, ReadMemory, AskSubagent) is described in text but would be easier to verify with explicit serialized examples in the supplement.
[Results: D0] The sentence 'The D0 behavior also has the structural signature predicted by direct acquisition' could be misunderstood as post hoc; it would help to cite the prespecified nature of the coloc direction earlier (it is mentioned in B.5, but the main text could make this explicit).

Circularity Check

1 steps flagged · score 3.0 of 10

One supporting detector validation is self-confirming, but the central GOLD-minus-SHAM behavioral contrast is an independent controlled experiment.

self definitional [Results, 'D0 Success Tracks the Acquired Target Value'; Controlled Artifact Construction; Supplementary B.5]
"The D0 behavior also has the structural signature predicted by direct acquisition. For Llama, coloc reaches AUROC 0.929 [0.881, 0.966] against earned CLEAN success and 0.908 [0.848, 0.956] against successful exposed SHAM trajectories. ... An accepted claim contains the target exactly once and directly resolves the question. GOLD fills the claim with the reference value."

coloc is defined as max_s r_q(s) r_a(s) over observed sources, with r_a using the model's predicted answer. Operational positives are GOLD trajectories that fail in CLEAN, succeed in GOLD, and expose a controlled source before the final answer. By construction that GOLD source contains the reference target and 'directly resolves the question,' so whenever the model succeeds (predicted answer = target) and the source is observed, r_a = 1 and r_q is high, forcing coloc high. The AUROC against earned CLEAN and SHAM successes is therefore a separation between sources built to contain the answer and sources built not to; the 'predicted' high-coloc signature is equivalent to the positive-label definition plus artifact construction, not an independent confirmation.

full rationale

The paper is a controlled empirical benchmark, not a formal derivation, so the circularity burden is low. The central estimands, Delta_GC = E[Y_G - Y_C] and Delta_GS = E[Y_G - Y_S], are paired contrasts measured on frozen HotpotQA populations; they are not derived from their own definitions, and the headline GOLD-minus-SHAM effects (19.1-25.9 pp in D0) are independent empirical outcomes. The main caveat is the SHAM control: the paper explicitly concedes that Delta_GS does not separately identify help from the correct value and interference from the incorrect value (Supplementary B.2) and that SHAM does not assert identical realized exposure or attention (Supplementary A.1). That is a control-validity limitation, not circularity. There are no load-bearing self-citations: the references are external benchmarks, model documentation, and prior contamination/attribution work, and no uniqueness theorem is imported from the authors' own prior work. The one genuinely construction-driven result is the D0 coloc AUROC: the positive label already requires an exposed GOLD source that contains the correct target and directly resolves the question, so high coloc is substantially forced by the artifact construction. The paper itself calls this a 'regime-specific validity check' rather than a general detector, which limits the severity. The D2 AUROC below 0.5 is likewise best read as a manipulation check of the D2 construction gate (max single-source construction score <= 0.25); the non-circular content there is the persistent positive GOLD-SHAM contrast. Overall, the central claim does not reduce to a fit or a self-citation chain; only a supporting detector validation is self-confirming, warranting a low but nonzero circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The central experiment is a controlled intervention, so there are no fitted free parameters in the headline contrasts. The main burden is the SHAM control assumption and the construction-specific gates for D2; both are stated explicitly. The invented entities are evaluation constructs with internal operational definitions, not external predictions.

assumptions (4)

domain assumption SHAM preserves source structure and exposure opportunity such that GOLD-SHAM isolates target correctness beyond exposure (Supplementary A.1).
The paper explicitly states SHAM does not assert identical realized exposure, attention, or subsequent action paths, so the assumption that GOLD-SHAM is a clean attribution of value correctness is load-bearing.
domain assumption The HotpotQA distractor development set and the frozen 135/38/36 qid subsets represent a meaningful evaluation population for the claims (Table S1).
The claims are conditioned on this narrow, shorter-answer subset; the paper is transparent, but the generalizability rests on this choice.
domain assumption The four action surfaces (retrieval, tool, memory, subagent) are exchangeable enough to be pooled under qid-clustered analysis (Table S2).
The paper treats them as controlled interface realizations, not independent systems; pooling assumes no surface-specific interaction that would alter the headline contrast.
ad hoc to paper The D2 two-source construction with max single-source construction score <= 0.25 ensures no single source answers the question (Supplementary A.5).
This is a proxy gate for distributed sufficiency; it is applied before outcomes are observed, but it is a construction-specific operationalization, not a demonstrated fact about information structure.

invented entities (2)

success provenance
purpose: New evaluation object: whether acquired target information supported the scored result.
This is a conceptual construct introduced by the paper; its operational handle is the GOLD-SHAM contrast, which is internal to the paper.
coloc score
purpose: Single-source question-answer co-location score used to test whether direct acquisition leaves a visible structural signature.
A new metric defined in the paper; it is a useful diagnostic but not independently validated outside this paper's settings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation." pith.science (2026). https://pith.science/paper/KUWAZ7XP

@misc{pith2026260724054,
  author       = {Pith},
  title        = {Pith review of: Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KUWAZ7XP}},
  note         = {Machine review of arXiv:2607.24054}
}

read the original abstract

A correct answer can conceal why an agent succeeded. Once agents change their information state during evaluation, correctness no longer distinguishes intended reasoning from answer acquisition. Outcome evidence and exposure detection do not establish whether success depended on an acquired target; we call this missing evaluation object success provenance. AcquaBench audits it through matched CLEAN, GOLD, and SHAM value substitution on four standardized surfaces with joint qid-clustered analysis. CLEAN retains benchmark-authorized information. GOLD makes the correct target available. SHAM preserves source structure and exposure opportunity but substitutes a matched incorrect value. GOLD minus CLEAN measures the total score response to correct-target availability; GOLD minus SHAM tests whether that response tracks target correctness beyond matched source exposure. In D0, GOLD exceeds SHAM by 19.1 to 25.9 percentage points, showing that success follows the correct value. In D2, GOLD still exceeds SHAM under distributed sufficiency while coloc no longer transfers as a high-score marker, with AUROC 0.376 and 0.142. Behavioral dependence can thus persist beyond this probe's intended observation unit. In model comparison, a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points without establishing rank inversion. Agent benchmarks should report success together with whether the evaluated information state supported it.

Figures

Figures reproduced from arXiv: 2607.24054 by the authors.

**Figure 2.** AcquaBench’s matched value-substitution audit. CLEAN contains only benchmark-authorized information; GOLD [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Behavioral value-substitution effects and single [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

[1]

obtain HotpotQA distractor development data under its original terms
[2]

WorkArena:HowCapableAreWebAgentsatSolving Common Knowledge Work Tasks? InProceedings of the 41stInternationalConferenceonMachineLearning,volume 235 ofProceedings of Machine Learning Research, 11642– 11662. PMLR. Es, S.; James, J.; Espinosa Anke, L.; and Schockaert, S
[3]

Association for Computational Linguistics

RAGAs: Automated Evaluation of Retrieval Aug- mentedGeneration.InProceedingsofthe18thConferenceof the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158. Association for Computational Linguistics. Gao,S.;andZhou,L.2026. CanAgentBenchmarksSupport Their Scores? Evidence-Supported Bounds for Interactive- Agent E...

arXiv 2026
[4]

InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 16022–16076

AppWorld: A Controllable World of Apps and Peo- ple for Benchmarking Interactive Coding Agents. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 16022–16076. Association for Computational Linguistics. Wang, J.; Ma, Z.; Li, Y.; Zhang, S.; Chen, C.; Chen, K.; and Le, X. 2024. GTA: A Benchmark for General Tool Agen...

arXiv 2024
[5]

extract scores and operational labels from the immutable trajectories
[6]

PeachtreeSt

Qwen2.5 Technical Report. arXiv:2412.15115. Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods inNaturalLanguageProcessing,2369–2380.Brussels,Bel- gium: Association for Computational Lin...

arXiv 2018
[7]

The screen- play for Alain Resnais’ second film is who?

compile the main paper and this supplement with the AAAI-27 submission style and render every PDF page for visual inspection. Model weights are not redistributed. The review package shouldincludesourcecode,configurationfiles,artifactmani- fests,sanitizedtrajectoryrecords,frozenderivedtables,anda manifest of hashes. Absolute server/workstation paths, user-...
[8]

resolvethethreepublicmodelcheckpointsandthefrozen runtime configuration

Show all 13 references

[9]

sample qids with seed 13, construct and freeze artifact manifests,andinspecttheconstructionauditbeforeinfer- ence
[10]

run the four action surfaces forClean,Gold, andSham with deterministic decoding and checkpoint each com- pleted cell
[12]

build paired triplets, resample qids for 5,000 replicates, and write the frozen JSON/CSV/TeX report artifacts
[2024]

InProceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, 8706–8719

Investigating Data Contamination in Modern Bench- marks for Large Language Models. InProceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, 8706–8719. Association for Computational Lingu...

2024
[2025]

arXiv:2504.12516

BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516. White, C.; Dooley, S.; Roberts, M.; Pal, A.; Feuer, B.; Jain, S.;Shwartz-Ziv,R.;Jain,N.;Saifullah,K.;Dey,S.;Agrawal, S.;Sandha,S.S.;Naidu,S.;Hegde,C.;LeCun,Y.;Goldstein, T.; Neiswanger, W.; a...

2025 arXiv

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.

[1] [1]

obtain HotpotQA distractor development data under its original terms

[2] [2]

WorkArena:HowCapableAreWebAgentsatSolving Common Knowledge Work Tasks? InProceedings of the 41stInternationalConferenceonMachineLearning,volume 235 ofProceedings of Machine Learning Research, 11642– 11662. PMLR. Es, S.; James, J.; Espinosa Anke, L.; and Schockaert, S

[3] [3]

Association for Computational Linguistics

RAGAs: Automated Evaluation of Retrieval Aug- mentedGeneration.InProceedingsofthe18thConferenceof the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158. Association for Computational Linguistics. Gao,S.;andZhou,L.2026. CanAgentBenchmarksSupport Their Scores? Evidence-Supported Bounds for Interactive- Agent E...

arXiv 2026

[4] [4]

InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 16022–16076

AppWorld: A Controllable World of Apps and Peo- ple for Benchmarking Interactive Coding Agents. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 16022–16076. Association for Computational Linguistics. Wang, J.; Ma, Z.; Li, Y.; Zhang, S.; Chen, C.; Chen, K.; and Le, X. 2024. GTA: A Benchmark for General Tool Agen...

arXiv 2024

[5] [5]

extract scores and operational labels from the immutable trajectories

[6] [6]

PeachtreeSt

Qwen2.5 Technical Report. arXiv:2412.15115. Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods inNaturalLanguageProcessing,2369–2380.Brussels,Bel- gium: Association for Computational Lin...

arXiv 2018

[7] [7]

The screen- play for Alain Resnais’ second film is who?

compile the main paper and this supplement with the AAAI-27 submission style and render every PDF page for visual inspection. Model weights are not redistributed. The review package shouldincludesourcecode,configurationfiles,artifactmani- fests,sanitizedtrajectoryrecords,frozenderivedtables,anda manifest of hashes. Absolute server/workstation paths, user-...

[8] [8]

resolvethethreepublicmodelcheckpointsandthefrozen runtime configuration