{"id":"578bec5c-4b9e-4025-b05b-e4c04201c64c","arxiv_id":"2506.03444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A logit-based method converts an LLM's numeric guesses into a calibrated prior over Pearson correlations and ranks expert-flagged hypotheses better than ranking by magnitude or by a fine-tuned RoBERTa classifier.","lead":"What if a language model could tell you which correlations in your data are actually surprising? This paper builds a distribution over correlation values from an LLM's token probabilities, and uses it to rank tens of thousands of real-world correlations so that expected relationships are deprioritized and unexpected ones are surfaced.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage and information-content evidence for calibration is fixed mostly by the fitted sigma=0.4, not by the logit distribution's shape; a mode-only baseline with the same sigma would likely reproduce them.","rationale":"The reader's weakest assumption identified the single fitted sigma and the calibration metrics; my concern is the same but sharper. The paper is a solid empirical study with real positive results that are independent of the calibration fit: mode-based sign accuracy (78.8%), MAE (0.26), and the Nexus retrieval precision are plausible and not invalidated by this concern. However, the word 'calibrated' in the central claim is carried by coverage and information content, and those metrics are dominated by the validation-selected sigma. Because KDE has identical sign accuracy but much lower coverage solely due to a smaller sigma, the comparison in Figure 3 already shows that coverage is mostly a bandwidth effect. The proposed mode+sigma baseline is a cheap, decisive check: if it matches LCP, the contribution is mode prediction plus a tuned smoother, not a calibrated logit-derived predictive distribution; if it does not match, the calibration claim survives. Either way the conditional acceptance is appropriate, so no verdict change is needed.","tokens_in":14540,"tokens_out":7287,"duration_ms":81527,"concrete_test":"Construct a 'mode+sigma' baseline: for each of the 2,096 benchmark pairs, take the LCP mode r_hat, form a point-mass prior delta(r - r_hat), smooth it with a Gaussian kernel of sigma=0.4 truncated to [-1,1], and recompute 95% credible interval coverage and average information content. If this baseline matches LCP within, say, 1 percentage point coverage and 0.03 nats information content, then the Section 4 calibration metrics are explained by the fitted bandwidth and mode location alone, and the calibration claim should be reframed or strengthened with a test that exercises the logit distribution's shape (e.g., proper scoring rules or interval calibration conditional on mode magnitude). As a transfer check, refit sigma on 10 different 300-pair validation subsamples and report coverage on the remaining pairs; large variation would indicate the single sigma is overfit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a 'calibrated' continuous distribution is supported primarily by 95% coverage (89.2%) and average information content (0.27) in Section 4. Both metrics are largely controlled by the single kernel width sigma=0.4, which is chosen in Section 2 by minimizing average negative log-likelihood at observed correlations on 300 validation pairs. With sigma=0.4, a Gaussian kernel covers most of [-1,1], so high coverage is almost a direct consequence of the tuning, and low information content follows from placing a smooth kernel near the mode (MAE 0.26). This is visible in the comparison with KDE: KDE has the same sign accuracy (78.8%) but coverage 59.9% because its Scott-rule sigma has median 0.08; simply widening KDE's kernels to sigma=0.4 would recover high coverage. Thus the reported coverage and IC do not independently demonstrate that the logit-derived mixture in Algorithm 1 is calibrated. The mode-based results (sign accuracy, MAE) are legitimate and do not depend on sigma, but they are predictive-accuracy results, not distributional-calibration evidence. A degenerate prior that places all mass on the LCP mode and smooths with sigma=0.4 would likely match the calibration numbers, in which case the shape of the logit distribution is incidental to the headline calibration claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Logit-based Calibrated Prior (LCP), a method that converts GPT-4o's token logits under a correlation-prediction prompt into a discrete distribution over [-1,1], then smooths it with a Gaussian kernel whose width sigma is tuned on a 300-pair validation set. The authors evaluate this prior on 2,096 real-world variable pairs, reporting sign accuracy 78.8%, MAE 0.26, 95% credible interval coverage 89.2%, and average information content 0.27, and compare against uniform, Gaussian, KDE, and fine-tuned RoBERTa baselines. They also apply LCP to rank expert-flagged correlations in the Nexus urban-data corpus and test context sensitivity with counterfactual scenarios.","tokens_in":14807,"tokens_out":9090,"duration_ms":95704,"significance":"The paper addresses a timely problem—automatic triage of discovered correlations—and the mode-based results (sign accuracy, MAE, and the Nexus retrieval experiment) are plausible and interesting. The benchmark construction is transparent and the anonymous repository is a strength. However, the headline calibration evidence is currently dominated by the single fitted kernel width sigma, so the central claim that LCP is a 'calibrated continuous predictive distribution' is not established. The paper would be significantly strengthened by a control that isolates the contribution of the logit distribution shape.","major_comments":[{"comment":"The coverage and information-content results are not independent evidence of calibration because both are controlled by the fitted kernel width. sigma*=0.4 is selected on 300 validation pairs by minimizing average negative log-likelihood, and the same sigma fixes the width of the 95% credible intervals and the density p(r_obs) used for information content. A point-mass prior at the LCP mode smoothed with the same sigma would likely achieve nearly the same 89.2% coverage and 0.27 information content, given MAE=0.26. The comparison with KDE (59.9% coverage with median sigma=0.08) demonstrates only that a larger kernel width increases coverage, not that the shape of the logit distribution matters. Please add a control that replaces the logit mixture with a delta at the mode while keeping sigma=0.4, and report coverage across confidence levels rather than a single 95% number.","section":"Section 2, Eq. (1); Section 4, Fig. 3"},{"comment":"The validation set is underspecified. The text says Dval consists of 300 randomly sampled correlations, disjoint from our evaluation dataset, but Section 3 only describes the 2,096-pair benchmark and does not mention a separate validation set. It is unclear whether these 300 pairs are drawn from the same Kaggle/Cause-Effect sources and whether they share tables or variable pairs with the evaluation set. Because sigma is the only fitted parameter and the paper claims it transfers to all pairs and to Nexus, the sampling process and the level of overlap must be documented.","section":"Section 2, validation set"},{"comment":"The claim that the prior 'generalizes to correlations not seen during LLM pretraining' is not supported by the contextual-contradiction experiment. That experiment uses 84 Cause-Effect pairs, replaces the context with a synthetic counterfactual generated by Gemini 2.5 Pro, and sets r_obs = -r_obs as an approximation. This tests whether predictions move in the expected direction, not whether the prior is accurate on unseen correlation values or domains. Please rephrase the claim or add a held-out evaluation based on a temporal or source-based split.","section":"Section 6 and Abstract"}],"minor_comments":[{"comment":"The value of k in the top-k decoding (line 5) is never stated; please report it and include a sensitivity analysis for the resulting discrete distribution.","section":"Algorithm 1"},{"comment":"The chi-square goodness-of-fit test treats the model-assigned probabilities p_j as pseudo-counts with M=1000, which is not a valid application of the chi-square test because these probabilities are not independent empirical frequencies; the normality-rejection claim should be supported by a different diagnostic or stated more cautiously.","section":"Appendix A"},{"comment":"The retrieval comparison is based on a single pool of 115 items with 15 targets; please report bootstrap confidence intervals or a significance test for the Precision@K and average-rank differences.","section":"Section 5, Table 1"},{"comment":"Please clarify which correlation value is used as r_obs when computing p(r_obs) for the Nexus ranking, since the text says ground-truth correlation values are unavailable.","section":"Section 5"},{"comment":"The numbers printed above the boxes in Figure 4 are not defined in the caption; please state what they represent (apparently standard deviations or mean values).","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the mode-based contribution is solid. The main risk is that the calibration claim is an artifact of sigma; asking for a delta-mode control and a calibration-curve analysis should settle it. I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The logit-based construction is a real contribution, and the mode-based results (sign accuracy, MAE) are believable. But the headline calibration numbers—89.2% coverage and information content 0.27—are mostly an artifact of the tuned kernel width σ=0.4, not evidence that the logit-derived discrete distribution is calibrated.\n\nWhat the paper does well: it shows how to turn raw token logits over a numeric field into a continuous predictive distribution by enumerating top-k token sequences, aggregating probabilities, and smoothing with a tuned Gaussian kernel. That is a practical and reusable technique for eliciting continuous priors from LLMs, and it does not require the model to parameterize a fixed distribution. The comparison against Gaussian and KDE priors is informative, especially the analysis of why Gaussian elicitation fails: GPT-4o defaults to σ=0.1, apparently because it assumes a sample size of 100. That is a useful empirical finding. The Nexus retrieval experiment, though small, demonstrates real utility: LCP ranks expert-flagged correlations higher than ranking by magnitude or by a fine-tuned RoBERTa. The contextual contradiction experiment is a creative way to probe memorization, and the inclusion of a multi-hop failure case shows honest reporting.\n\nThe main soft spot is the calibration evidence. The paper tunes σ on 300 validation pairs to minimize negative log-likelihood, then reports coverage and information content on the evaluation set with that same σ. A prior that placed all of its mass at the LCP mode and smoothed with the same σ=0.4 would likely produce similar coverage and IC. So the wide kernel, not the shape of the logit distribution, is doing the calibration work. That does not invalidate the mode-based results, but it means the paper's claim of a \"calibrated continuous predictive distribution\" is overstated. The validation set's origin is also unspecified—it says 300 pairs disjoint from evaluation, but not where they are drawn from. That matters because the tuned σ carries the calibration story. The memorization probe is on a selected subset (84 pairs where the initial sign prediction was correct), so the 95.2% under contradictory contexts is encouraging but not a clean generalization result.\n\nOverall, the core technical idea and the predictive-accuracy results are solid and worth publishing. The calibration claims should be softened and re-framed in revision, and the validation split should be documented. I would send this to peer review; it is the kind of paper that will elicit useful discussion about what \"calibration\" means for LLM-elicited priors.","headline":"The logit-based prior is a real and reusable technique; the mode-based results are credible, but the 'calibrated' headline is mostly a tuned kernel width, not the logit distribution.","tokens_in":15321,"tokens_out":3058,"would_cite":true,"duration_ms":32530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A calibrated prior built from an LLM's raw logits predicts which real-world correlations deserve a second look.","keywords":["hypothesis assessment","correlation prior","LLM logits","calibration","surprise ranking","prior elicitation","Pearson correlation","data discovery"],"falsifier":"Compute LCP's 95% credible interval coverage on a fresh set of variable pairs from an unseen domain without retuning $\\sigma$; if coverage falls far below the reported 89.2% (for example, below 80%), the transfer assumption fails. As a second check, if ranking a new pool of expert-flagged hypotheses by LCP likelihood does not beat magnitude-based and random rankings, the central utility claim is falsified.","tokens_in":14321,"feed_emoji":"📈","tokens_out":6955,"duration_ms":77204,"temperature":0.7,"pith_summary":"The paper tries to show that an LLM's pretrained knowledge can be turned into a usable statistical prior for hypothesis assessment: given two variables and their context, the prior says what correlation an informed observer would expect, so an observed correlation far from that expectation counts as surprising and worth expert attention. It proposes the Logit-based Calibrated Prior (LCP), which reads the model's raw token logits when it answers a prompted correlation question, assembles the competing numeric answers into a discrete distribution, and smooths it into a continuous density over $[-1,1]$. On 2,096 real-world variable pairs LCP reaches 78.8% sign accuracy, mean absolute error 0.26, and 89.2% coverage of its 95% credible intervals, while cutting the average information content of observed correlations from 0.69 to 0.27. The authors also argue the prior generalizes to counterfactual contexts, so it reasons from context rather than memorized name pairs.","feed_headline":"LLM logits rank surprising correlations at 78.8% accuracy","feed_subtitle":"A calibrated prior surfaces expert-worthy urban data correlations while down-ranking trivial ones.","key_machinery":"The load-bearing object is the logit-derived discrete distribution over correlation values: at each decoding step of a structured numeric answer, the method keeps the top-$k$ token probabilities, enumerates all token sequences that form valid floats in $[-1,1]$, sums their log-probabilities, merges duplicate numeric values, and softmax-normalizes. This discrete distribution is then smoothed into the LCP density $f(r) = \\frac{1}{Z} \\sum_{j=1}^N p_j \\mathcal{N}(r; r_j, \\sigma^2)$ truncated to $[-1,1]$, with $\\sigma = 0.4$ chosen by negative log-likelihood on a validation set. This machinery matters because it converts the LLM's raw belief state into a density $p(r \\mid C_{X,Y})$ evaluable at any observed correlation, which is exactly what surprise scoring, information-content measurement, and credible-interval calibration require.","core_discovery":"The central discovery claimed is that a non-parametric, logit-derived distribution, not a Gaussian or other parametric form, faithfully captures what an LLM believes about a correlation, and that once calibrated it behaves like a genuine predictive distribution for real data. LCP's construction aggregates the joint log-probabilities of all top-$k$ token sequences that parse to valid numbers in $[-1,1]$, merges sequences that map to the same value, and softmax-normalizes them; the resulting discrete prior is smoothed with truncated Gaussian kernels of width $\\sigma = 0.4$, tuned on 300 validation pairs to minimize average negative log-likelihood. With that single width, the prior assigns 89.2% coverage to 95% credible intervals, outperforms both parametric and uncalibrated KDE baselines, beats a fine-tuned masked-language-model classifier on thresholded binary correlation prediction despite requiring no training itself, and ranks expert-flagged correlations from noisy urban data above magnitude-based or classifier-based rankings. The contextual-contradiction experiment, in which contrived contexts reverse 84 known relationships, reports 95.2% sign accuracy under the reversed priors, which the authors take as evidence of reasoning rather than memorization.","pith_inferences":["If LCP's success reflects general world-knowledge calibration, the same logit-aggregation trick could elicit priors for other scalar relationship statistics, such as trend slopes, effect sizes, or risk ratios, wherever an LLM can answer a structured numeric prompt.","The single fixed $\\sigma = 0.4$ is a strong transfer assumption; a natural extension the paper does not test is hierarchical or input-dependent kernel widths that adapt uncertainty to variable-pair familiarity.","The surprise score is relative to the LLM's knowledge, not to a domain expert's knowledge; in specialized fields an LLM may systematically call familiar relationships surprising, so field-specific recalibration is a testable next step.","The benchmark's variable pairs come from public tabular and urban open data; applying the same pipeline to scientific or medical variable pairs would test whether the coverage and ranking gains persist outside those distributions."],"forward_implications":["Surprise can be scored as $-\\log p(r_{\\mathrm{obs}})$, so a discovery system can rank thousands of variable pairs by how much they deviate from an informed prior rather than by $|r|$.","A single prior supports any correlation threshold without retraining, unlike a binary classifier that must be re-fit for each decision boundary.","The retrieval result in noisy urban data implies that LCP can surface expert-validated hypotheses even after joins, aggregation, and missing-value transformations.","The contextual-contradiction result implies that the prior updates with modified contexts, making it applicable to variables whose relationships are not in the pretraining corpus."],"supporting_citations":[{"why":"Supplies the fine-tuned masked-language-model classifier baseline and the binary correlation-prediction setup LCP is compared against.","marker":"[30]"},{"why":"Supplies the 40,538 pairwise correlations from urban open data and the expert-flagged evaluation pool used for retrieval ranking.","marker":"[8]"},{"why":"Provides the bike-station density and community-wealth hypothesis that serves as one expert-flagged retrieval target.","marker":"[7]"},{"why":"Cause-Effect Pairs contributes the 96 significant variable pairs and the basis for the contextual-contradiction test.","marker":"[15]"},{"why":"Provides the Gaussian prior elicitation method that serves as the main parametric baseline.","marker":"[1]"},{"why":"Identifies the LLM whose token logits the method uses; all priors in the evaluation are elicited from this model.","marker":"[17]"},{"why":"Supplies the urban datasets behind the retrieval evaluation.","marker":"[19]"},{"why":"Supplies the chi-square goodness-of-fit test used to reject the Gaussian shape of LLM output distributions, justifying the non-parametric construction.","marker":"[28]"}],"fun_headline_variants":["Logit prior outscores tuned classifier on correlations","LLM logit prior predicts correlation novelty without training","Surprising correlations found by calibrated LLM prior","LLM prior beats classifier on correlation ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the token probabilities the LLM assigns to candidate numeric answers are a faithful expression of its belief, and if the single smoothing width $\\sigma = 0.4$ tuned on 300 validation pairs transfers that calibration to every other variable pair and to the retrieval task.","fun_headline_variants_meta":{"raw":{"variants":["Logit prior outscores tuned classifier on correlations","LLM logit prior predicts correlation novelty without training","Surprising correlations found by calibrated LLM prior","LLM prior beats classifier on correlation ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1990,"prompt_tokens":1093,"completion_tokens":897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":838}},"tokens_in":709,"tokens_out":897,"duration_ms":10680,"temperature":1.0,"reasoning_tokens":838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:03:13.292692+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute LCP's 95% credible interval coverage on a fresh set of variable pairs from an unseen domain without retuning $\\sigma$; if coverage falls far below the reported 89.2% (for example, below 80%), the transfer assumption fails. As a second check, if ranking a new pool of expert-flagged hypotheses by LCP likelihood does not beat magnitude-based and random rankings, the central utility claim is falsified.","supporting_citations":[{"cited_title":"Can large language models predict data correlations from column names? Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the fine-tuned masked-language-model classifier baseline and the binary correlation-prediction setup LCP is compared against."},{"cited_title":"Nexus: Correlation discovery over collections of spatio-temporal tabular data.Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the 40,538 pairwise correlations from urban open data and the expert-flagged evaluation pool used for retrieval ranking."},{"cited_title":"Riding tandem: Does cycling infrastructure investment mirror gentrification and privilege in portland, or and chicago, il? Research in Transportation Economics, 60:14–24, 2016","cited_arxiv_id":null,"evidence_quote":"Provides the bike-station density and community-wealth hypothesis that serves as one expert-flagged retrieval target."},{"cited_title":"Distinguishing cause from effect using observational data: methods and benchmarks.Journal of Machine Learning Research, 17(32):1–102, 2016","cited_arxiv_id":null,"evidence_quote":"Cause-Effect Pairs contributes the 96 significant variable pairs and the basis for the contextual-contradiction test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies the LLM whose token logits the method uses; all priors in the evaluation are elicited from this model."},{"cited_title":"Chicago data portal, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the urban datasets behind the retrieval evaluation."},{"cited_title":"Statistical methods, 8thedn.Ames: Iowa State Univ","cited_arxiv_id":null,"evidence_quote":"Supplies the chi-square goodness-of-fit test used to reject the Gaussian shape of LLM output distributions, justifying the non-parametric construction."}],"review_version":1}