{"id":"3133aacf-8271-497c-b639-791178e3564e","arxiv_id":"2508.05469","paper_version":4,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"An AI evaluation method that estimates mutual information through total variation distance stays robust under adversarial attack (AUC 0.70-0.77) where direct quality judgments decay to chance, all without ground truth.","lead":"This paper proposes a new way to evaluate AI systems without correct answers to compare against: instead of asking whether an output is good, it measures how much information the system's outputs carry. If it holds up, it could make AI evaluation much harder to cheat, which matters as AI models increasingly judge other AI models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theoretical TVD-MI guarantees require an estimator that the prompt-based LLM surrogate is not shown to be; without calibration evidence, the AUC claim does not inherit the polynomial robustness proof.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the jump from a theoretical f-divergence estimator to a prompt-based LLM surrogate is unestablished. This is not a disagreement with the paper's internal logic; the abstract is coherent as far as it goes. But the central claim (that prompting the same system for information relationships yields robust evaluation) depends on the LLM's outputs being a valid TVD estimate. Without full text, we cannot see whether the authors provide calibration or correlation analyses. The concern is therefore not a refutation but a specific empirical premise that must hold. Given the abstract-only review, the appropriate verdict remains UNVERDICTED with low confidence, exactly as the reader concluded. My concrete test would settle whether the premise holds, but it requires access to the full paper's experiments or the pre-registration. No additional objection beyond the reader's is needed; the concern is sufficiently precise and load-bearing.","tokens_in":828,"tokens_out":2056,"duration_ms":22914,"concrete_test":"On the pre-registered datasets (or on synthetic tasks where true distributions are known), compute the ground-truth TVD between the agent outputs (or between output and reference) and compare it with the LLM's prompted TVD estimates. If the Spearman correlation is below ~0.5 or calibration is poor, the surrogate does not implement the theoretical estimator. Additionally, re-run the same evaluation with a different LLM and a rephrased prompt; if the AUC shifts by more than 0.1, the headline robustness is not a stable property of TVD-MI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that prompting an LLM to estimate information relationships yields a robust evaluation metric with polynomial guarantees under adversarial attacks. But the theoretical result about f-divergence estimators (e.g., TVD) is established for a statistical estimator that has access to samples from the relevant distributions. The actual method replaces this with an LLM's prompted judgments, and the abstract provides no evidence that these prompted outputs are a faithful or calibrated approximation of TVD. If the LLM's responses carry prior biases, fail on high-dimensional dependencies, or even systematically deviate from the true TVD, the polynomial guarantee does not transfer. The AUC range 0.70–0.77 is moderate, and without a calibration check against ground-truth TVD, it remains possible that the reported robustness is an artifact of the specific model/prompt rather than of TVD-MI itself. This gap is load-bearing because the paper's theoretical motivation is the only reason to expect the empirical method to resist attacks; if the surrogate does not track the statistic, the mechanism collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an AI evaluation method that avoids ground truth by prompting an overseer model to estimate information-theoretic quantities, specifically total variation distance (TVD), as a proxy for mutual information between agent behavior and some target variable. The authors argue that, unlike direct quality judgments, this 'mutual evaluation' approach is robust to adversarial manipulation, citing polynomial sample guarantees for TVD estimation in contrast to exponential barriers for worst-case MI estimation. The abstract reports empirical AUC values of 0.70--0.77 under adversarial attacks, with other approaches decaying toward chance, and claims that truthful reporting becomes optimal. A pre-registration link is provided.","tokens_in":1026,"tokens_out":1536,"duration_ms":18704,"significance":"If the claimed result holds, the paper offers a principled alternative to current LLM-as-judge evaluation methods, with formal robustness guarantees grounded in information theory. This is potentially significant for benchmarking AI systems without ground truth, especially in adversarial settings. The paper's strengths include an explicit connection to established information-theoretic results, a falsifiable empirical claim, and a pre-registration, all of which are commendable. However, the abstract alone leaves the central mechanistic link between the theoretical guarantees and the prompt-based surrogate unspecified, so the significance is conditional.","major_comments":[{"comment":"The central claim—that prompting an LLM to estimate information relationships yields a metric with polynomial adversarial guarantees—requires that the prompted outputs faithfully approximate the TVD estimator for which the guarantees are proven. The abstract provides no calibration evidence, theoretical or empirical, that an LLM's prompted judgments track true TVD. If the LLM surrogate is biased or systematically deviates, the polynomial guarantee does not transfer. This is load-bearing: the theoretical motivation is the only stated reason to expect robustness. The authors should state the exact estimator assumptions in the theorem and provide validation on tasks with known ground-truth TVD, such as controlled synthetic distributions or labeled proxy examples.","section":"Abstract"},{"comment":"The empirical claim 'area under the curve 0.70--0.77' is reported without any experimental protocol: dataset, model, attack model, baselines, number of runs, or error bars. The comparison 'other approaches can decay toward chance' is likewise unspecified. Without knowing which baselines were used and how variability was quantified, the reported robustness cannot be assessed. The full text should include a detailed experimental section with confidence intervals and a table of baselines across multiple attack strengths.","section":"Abstract"},{"comment":"The phrase 'making truthful agent reporting an optimal strategy' suggests that truthfulness is a design property of the incentive mechanism. If the overseer's prompting strategy is assumed to be faithful rather than derived, the optimality claim is circular relative to the information-theoretic guarantee. The authors should clarify the game-theoretic equilibrium notion used, specify the overseer's strategy set, and state whether the optimality theorem treats the prompt-based estimator as an exogenous oracle or as a strategic player whose incentives are analyzed.","section":"Abstract"}],"minor_comments":[{"comment":"The abbreviation 'TVD-MI' is used without a definition; the reader must infer that it means total-variation-distance-based mutual information estimation. A brief definition would improve clarity.","section":"Abstract"},{"comment":"The phrase 'polynomial guarantees under attack' is vague: polynomial in what parameter (sample size, dimension, number of agents), and under what attack model (adversarial perturbation of samples vs. strategic agent responses)? The full text should specify the theorem precisely.","section":"Abstract"},{"comment":"The pre-registration URL is given but no results are linked; consider including a note on whether the reported outcomes match the pre-registered analysis plan.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review, so the major comments necessarily target the gap between the theoretical claim and the operational surrogate. If the full paper provides a theorem with an explicit estimator and a calibration study, the central claim is likely defensible. The reported AUC range is moderate, so the authors should also ensure that their baselines are not artificially weak; a strong baseline such as direct LLM quality judgments with the same model and prompt budget is essential. The pre-registration is a positive sign, but the editor may wish to check that the full text adheres to the stated plan."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If the full paper backs up the abstract, this is a real step forward for ground-truth-free evaluation. The new idea is to use total variation distance as a proxy for mutual information and then prompt the overseer to estimate that information relationship, instead of asking for a quality judgment. The move is clever: TVD has nicer worst-case properties than MI, and the strategic framing gives a mechanism-design justification for why truthful reporting should be optimal. The pre-registration link is also a genuine strength; it suggests the authors are committed to being accountable for their empirical claims.\n\nThe empirical claim—AUC 0.70–0.77 under attack while other methods decay toward chance—is concrete and falsifiable. But the abstract alone does not let you verify it. No datasets, no baselines, no error bars, no protocol. That is an information limitation, not necessarily a flaw in the paper.\n\nThe soft spot that matters is the one the stress-test flags: the polynomial guarantees are proven for a statistical estimator that has access to samples from the relevant distributions. The actual method replaces that with an LLM's prompted judgments. Unless the authors show that the LLM's outputs are a calibrated surrogate for TVD, the theory does not transfer. If the LLM carries its own biases or fails on high-dimensional dependencies, the robustness you see empirically could be an artifact of the specific model and prompt, not of TVD. That gap is load-bearing because the theory is the only reason to expect the method to generalize. I hope the full text includes a calibration check against ground-truth TVD; if it does, this could be a solid contribution.\n\nThe reader's low confidence and abstract-only verdict are appropriate. I would not cite this from the abstract alone, and I would want to see the full text before bringing it to a reading group. But this deserves a serious referee: the idea is original, the pre-registration is a good sign, and the claims are sharp enough to be checked. The referee's main job should be to press on the surrogate-calibration issue and the experimental protocol.\n\nRecommendation: send it to review, not desk reject. If the full text closes the gap, it will be a useful paper for anyone working on LLM-as-judge pipelines, red-teaming, or self-improvement loops.","headline":"A promising abstract linking TVD robustness to prompted MI estimation, but the bridge from theory to the LLM surrogate is unshown and load-bearing; worth a referee if the full text closes that gap.","tokens_in":1505,"tokens_out":1321,"would_cite":false,"duration_ms":14993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that evaluating AI systems by prompting an overseer model to estimate information relationships—specifically total variation distance—is robust to adversarial manipulation, with empirical AUC 0.70–0.77 where direct quality","keywords":["AI evaluation without ground truth","adversarial robustness","mutual information","total variation distance","f-divergences","prompting strategies","peer prediction","strategic gaming"],"falsifier":"A controlled experiment where an adversary modifies agent outputs such that the true mutual information with the ground-truth state stays fixed but the prompted overseer's TVD-MI estimates drop to chance would falsify the claim that prompting is a faithful surrogate. Conversely, if the prompted estimates continue to track true information structure under attack, the claim survives.","tokens_in":700,"feed_emoji":"📏","tokens_out":3140,"duration_ms":32214,"temperature":0.7,"pith_summary":"The paper tries to establish a ground-truth-free way to evaluate AI systems: instead of asking a model to judge quality directly, an overseer model is prompted to estimate information relationships—how much one response tells you about another—treating the agent being evaluated as a strategic player. It argues that strategic gaming corresponds to information loss, so measuring a particular distance called total variation distance (TVD) gives a statistic (TVD-MI) that keeps working under adversarial attacks where direct judgment methods fall to chance. The claims are backed by a theoretical analysis showing TVD-MI has polynomial sample guarantees under attack, whereas plain mutual information estimation has an exponential worst-case barrier, plus empirical results (AUC 0.70–0.77) under adversarial attacks. If true, this would give AI evaluators a principled, attack-resistant alternative to human-labeled benchmarks.","feed_headline":"Measure AI by information loss, not quality vibes","feed_subtitle":"Prompting an overseer to estimate TVD-MI keeps detection working where direct quality judgments fall to chance.","key_machinery":"The central object is the total variation distance (TVD), an f-divergence that measures the absolute difference between two probability distributions. It is used to define a mutual-information-like statistic (TVD-MI) that the paper shows has polynomial sample-complexity guarantees even under adversarial manipulation, unlike plain mutual information which faces an exponential worst-case barrier. The prompting step makes this theoretical estimator operational: the overseer model is prompted to estimate information relationships, turning the theoretical guarantee into a practical evaluation tool.","core_discovery":"The central claim is that strategic gaming in AI evaluation is equivalent to information loss, and that this equivalence can be exploited to evaluate systems without ground truth. The authors propose 'mutual evaluation,' where the overseer is a strategic player estimating mutual information by prompting, making truthful reporting an optimal strategy. They show that certain f-divergences, particularly total variation distance, maintain polynomial guarantees under adversarial attack, building on an established exponential barrier for worst-case mutual information estimation. Empirically, under adversarial attacks, the TVD-MI estimator retains effectiveness with AUC 0.70–0.77 while other approa","pith_inferences":["Editorial inference: Because total variation distance bounds many other f-divergences, TVD-MI robustness might extend beyond adversarial prompt attacks to natural distribution shifts; a testable extension is to compare TVD-MI against direct judgments under domain shift or noise.","Editorial inference: If the overseer and the agent are the same model, the method becomes a self-evaluation loop; varying the relative strength of the overseer versus the agent would reveal when the information estimate begins to break down.","Editorial inference: The item-level decomposition could be applied to peer-review-like settings to identify which submissions carry genuine signal, a natural extension of the paper's claim about overcoming a peer-prediction limitation."],"forward_implications":["AI systems can be evaluated without gold labels by prompting the same model to measure information relationships, reducing dependence on human-annotated ground truth.","TVD-MI-based evaluation is more resistant to adversarial prompt attacks than direct quality judgments, making it a candidate for red-teaming and safety evaluation.","Pairwise comparisons can be converted into item-level detection scores without ground truth, enabling more granular and reliable signal extraction than standard peer prediction.","The polynomial guarantee for TVD-MI provides a theoretical reason to choose f-divergences, rather than plain mutual information, when designing evaluation metrics."],"supporting_citations":[],"fun_headline_variants":["Information loss reveals AI gaming, no ground truth needed","Mutual evaluation: prompt for information, not quality scores","TVD-MI keeps AI checks robust where quality judgments fail","Strategic gaming in AI is information loss—exploit it","Truthful AI reporting becomes optimal via mutual information"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that prompting an overseer model to estimate information relationships yields a faithful surrogate of the idealized TVD-MI statistic, so that the polynomial robustness guarantees proven for TVD-MI transfer to the prompted estimates.","fun_headline_variants_meta":{"raw":{"variants":["Information loss reveals AI gaming, no ground truth needed","Mutual evaluation: prompt for information, not quality scores","TVD-MI keeps AI checks robust where quality judgments fail","Strategic gaming in AI is information loss—exploit it","Truthful AI reporting becomes optimal via mutual information"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":980,"prompt_tokens":675,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":419,"tokens_out":305,"duration_ms":3556,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:18:46.588233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment where an adversary modifies agent outputs such that the true mutual information with the ground-truth state stays fixed but the prompted overseer's TVD-MI estimates drop to chance would falsify the claim that prompting is a faithful surrogate. Conversely, if the prompted estimates continue to track true information structure under attack, the claim survives.","supporting_citations":[],"review_version":2}