{"id":"c0281e20-06dc-4341-90d4-958047392d83","arxiv_id":"2412.00546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reranking symmetric-task inputs so that query-relevant elements occupy high-exposure positions can substantially reduce LLM errors on aggregate queries.","lead":"This paper proposes reranking the elements of an unordered input, such as rows of a database table or edges of a graph, before sending them to a large language model, placing query-relevant elements in positions the model is least likely to overlook.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The learned exposure function is transferred from token counting to graph/DB tasks without a direct validation; Section 6.2's claimed cross-task consistency is asserted but not demonstrated.","rationale":"I read the paper as proposing a black-box wrapper that reranks symmetric-task inputs using (a) a learned position-exposure function and (b) helper-estimated relevance scores. The bipartite debiasing idea is plausible and the Sinkhorn connection is a useful observation; the convergence proof is informal but not the main issue. The load-bearing step is that a single exposure curve, learned on a synthetic token-counting task, is applied to graph and database tasks. The paper's own related-work section (6.2) cites Machlab et al. showing that in-context recall is content- and structure-dependent, which makes the transfer assumption non-obvious. The only direct evidence for transfer is Figure 4, which is a two-dataset, one-model ablation without error bars. A task-specific exposure might change which positions are preferred, which would change both the optimal ordering and the reported proximity. The reader's weakest assumption identifies exactly this gap; I agree. I would not change the verdict: the required check is straightforward and the method could survive it, so CONDITIONAL remains appropriate. Secondary concerns (zero-error optimum assumption in Table 2, cherry-picked 99% figure, missing error bars) reinforce the conditionality but are not the single most load-bearing issue.","tokens_in":19790,"tokens_out":13855,"duration_ms":125127,"concrete_test":"Run the Section 5.1 window-placement exposure-discovery procedure directly on the IMDB and Graph Degree tasks for GPT-4o Mini: for each of the 9 relative-position bins, place the single query-relevant element (the edge incident to the queried node; the row satisfying the aggregation predicate) in that bin amid non-relevant elements, and measure the LLM error over at least 100 runs per bin. Compare the resulting exposure vector to the token-counting exposure used in the paper using Spearman rank correlation; if rho < 0.8 or the argmax bin shifts by more than one bin, the transfer assumption fails. Re-run Table 2 with the task-specific exposure to quantify the degradation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the exposure function learned on a token-counting task (Section 5.1) being valid for graph-degree and database-aggregation tasks (Tables 2a/2b, Figure 4). This transfer is asserted, not measured. The paper never applies the Section 5.1 window-placement procedure to a downstream task to obtain a task-specific exposure curve. Figure 4 is the only transfer evidence, and it reports a single LLM (GPT-4o Mini), two datasets, no variance, and no comparison to a task-specific exposure baseline. Moreover, Section 6.2 cites Machlab et al. [28] for the opposite conclusion—recall is 'heavily dependent on the structure and content of the prompt'—and the paper's rejoinder, 'our experiments demonstrate that this recall pattern is consistent across different tasks within the same category,' is exactly the missing experiment. On top of this, Eq. 9 (1/E[epsilon] ∝ (1/n) Sum X_L(i)·Rel_q(t_i)) is an unproven functional form: expected error is a sum of miss probabilities, so its reciprocal is not generally linear in the exposure-weighted relevance sum. If the transfer fails or the linear model is miss-specified, the reranking can be no better than relevance-only sorting, and the 99% proximity figure is unearned.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies symmetric tasks—queries over an unordered bag of elements—and proposes to improve LLM accuracy by reranking the input before the LLM is invoked. The proposed pipeline is two-stage: offline, the paper estimates a position-dependent \"exposure\" function X_L(i) for the target LLM, and online it estimates query-dependent relevance scores for each input element using a smaller helper LLM. Relevance estimation is formulated as a bipartite debiasing problem and solved by an iterative Sinkhorn-style matrix scaling, with Theorem 1 claiming convergence. Experiments cover a synthetic graph-degree task and three real-world database aggregation tasks, using GPT-3.5 Turbo and GPT-4o Mini as target LLMs and five open-weight models as helpers. The paper reports ranking utilities approaching the optimal ordering and output errors close to a normalized optimum, with the abstract claiming up to 99% proximity to the optimal upper bound.","tokens_in":20046,"tokens_out":6435,"duration_ms":64651,"significance":"If the central claims hold, the paper offers a useful black-box wrapper for symmetric data-management tasks: it requires no access to the target LLM's internals and only a cheap helper model at query time. The bipartite debiasing formulation is a clean adaptation of matrix scaling to pointwise relevance estimation, and the convergence argument is essentially sound modulo a support-condition detail. The problem formulation is clear and the experimental scope (two commercial LLMs, five helper models, four task settings) is reasonable for a first study. However, the main intellectual load is carried by an unvalidated exposure model and an asserted transfer of exposure learned on a synthetic token-counting task to downstream tasks. The paper currently does not establish either the functional form in Eq. (9) or the cross-task consistency claimed in Section 6.2, so the abstract's 99% proximity figure is not yet backed by the evidence.","major_comments":[{"comment":"The exposure model 1/E[ε] ∝ (1/n) Σ X_L(i) · Rel_q(t_i) is stated without derivation or validation. Expected error is naturally a sum of per-element miss probabilities, and the reciprocal of such a sum is not generally linear in an exposure-weighted relevance sum. Since this equation is the basis for the least-squares estimator in Eq. (10) and for the entire reranking utility in Eq. (2), it is load-bearing. The paper should either derive this form from a concrete generative model of LLM misses, or validate it empirically by comparing predicted inverse errors with observed inverse errors on held-out permutations. Without that, the estimated X_L may be an artifact of the assumed functional form rather than a meaningful position-exposure curve.","section":"Section 4, Eq. (9)"},{"comment":"The transfer of exposure from the synthetic token-counting task in Section 5.1 to graph-degree and database tasks is asserted but not demonstrated. Figure 4, which is the only direct transfer evidence, reports GPT-4o Mini, two datasets, no variance or error bars, and no comparison against a task-specific exposure baseline. Section 6.2 claims \"our experiments demonstrate that this recall pattern is consistent across different tasks within the same category,\" but no experiment in the paper measures the recall pattern on a downstream task; the cited Machlab et al. [28] actually reports that recall is heavily dependent on prompt structure and content. A direct validation is needed: estimate exposure on each downstream task (e.g., by the window-placement procedure) and compare reranking with transferred exposure, task-specific exposure, and no exposure. This is essential because the abstract's improvement claim depends on the transferred exposure being valid.","section":"Section 5.5 and Section 6.2"},{"comment":"There is a mismatch between the proposed estimator and the experimental exposure discovery. Section 4.1 derives a least-squares solution for X from Eq. (10), but Section 5.1 instead measures exposure as the inverse of the average error when relevant tokens are placed in windows, and no confidence intervals from Eqs. (12)–(13) are reported. Additionally, Table 1 computes ranking utility using an arbitrary exposure X_L(i)=1/i, not the learned exposure, while Table 2 normalizes errors so that Random is 1.00 and Optimum is 0.00 in every subtable. The abstract's \"up to 99% proximity to the optimum\" therefore appears to refer to the ranking-utility metric with a fixed 1/i exposure, not to the final accuracy improvement obtained with the learned exposure. The paper should report absolute errors, state explicitly whether Table 2 uses the learned exposure or relevance-only sorting, and separate the claims about relevance estimation from the claims about exposure transfer.","section":"Section 5.1 and Tables 1–2"},{"comment":"The convergence proof identifies the iterative updates with Sinkhorn's algorithm, but the stated support condition (\"at least one positive diagonal\") is not sufficient for Sinkhorn--Knopp convergence; the support of W must be fully indecomposable (or at least have total support). This is likely satisfied for random shuffles but should be stated correctly. More importantly, the bias model in Eq. (3) assumes that each evaluation E_j has a single multiplicative bias β_j shared by all elements in that partition. This is a strong assumption that is not tested, and the fixed point of the scaling is only meaningful if the assumed bias structure holds. The authors should at least show a robustness check, e.g., correlation between the debiased scores and ground-truth relevance on a labeled subset.","section":"Theorem 1 and Section 3.2"}],"minor_comments":[{"comment":"The axis labels contain typos: \"Relative Posi ion in Promp \" and \"Expos re\" should be \"Relative Position in Prompt\" and \"Exposure\".","section":"Figure 3"},{"comment":"The figure reports single-point errors without error bars or statistical significance tests, although each value is stated to be an average of 10 runs; adding standard deviations or confidence intervals would materially strengthen the transfer claim.","section":"Section 5.5, Figure 4"},{"comment":"The paper states that the code is \"accessible through this repository\" but does not provide a URL; a working link should be included for reproducibility.","section":"Section 5 experimental setup"},{"comment":"The normalization procedure is underspecified: \"errors are normalized for each helper model to align them in the interval [0,1]\" does not say whether the normalization preserves absolute differences or forces Random and Optimum to 1.00 and 0.00; the original, unnormalized errors should be reported in addition.","section":"Tables 2a/2b"},{"comment":"The confidence-interval analysis in Eqs. (11)–(13) is not used anywhere in the experiments; reporting the estimated variances for the exposure curves in Figure 3 would connect the theory to the measurements.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a timely and potentially useful idea, and the bipartite debiasing component is a clean contribution. The main risk is that the central exposure-transference claim is asserted rather than demonstrated, and Eq. (9) is an untested linear model. I would encourage the editor to request a revision that adds (a) a direct validation of Eq. (9), (b) a transfer experiment comparing task-specific versus transferred exposure, and (c) clear reporting of absolute errors. With those additions, the paper could be a solid contribution to the SIGMOD-Companion audience; without them, the 99% proximity claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper addresses a real, widely observed problem: LLMs miss elements in long symmetric inputs, and since order is semantically irrelevant for these tasks, reranking is a legitimate free variable. The formulation is new as far as I know: learn position exposure, estimate query relevance through a helper LLM, and combine them into a ranking utility. The bipartite debiasing step is the most solid piece. Modeling helper scores as biased evaluations and solving via Sinkhorn scaling is correct, and the convergence argument, though abbreviated, is the standard matrix-scaling result. That part deserves credit.\\n\\nThe experiments are broad for a short paper: two target LLMs, five helper models, and four task/dataset settings. When the helper is competent, the reranking consistently approaches the optimal ordering and reduces output error; the poor Qwen2 cases are reported rather than hidden. That honesty helps.\\n\\nThe soft spots are real but not fatal. Equation 9 is load-bearing and asserted, not derived: inverse error proportional to an exposure-weighted relevance sum is a heuristic. More importantly, the exposure function is learned on a token-counting task and then applied to graph edges and table rows without a direct check that the exposure curve transfers. Figure 4 is the only transfer evidence: one LLM, two datasets, no error bars, no task-specific exposure baseline. The sentence in Section 6.2 claiming that experiments demonstrate consistency across tasks is exactly the missing experiment, especially since Machlab et al., which they cite, says recall is prompt-dependent. I would not call the pipeline circular: exposure is fit on calibration tasks and evaluated on held-out tasks. But the transfer is unvalidated.\\n\\nThe lack of variance and statistical tests, plus the per-helper normalization, makes the \\\"99% proximity\\\" headline a best-case number rather than a typical result. Code is claimed but I could not find a working link in the text.\\n\\nOverall, the core idea holds up as a plausible model-agnostic wrapper, and the ranking utility results show that relevance estimation works. What is not yet shown is that exposures learned on one synthetic task generalize across tasks. That gap is fixable with a direct transfer experiment and honest error bars. I would send this to peer review and require those additions, but I would not desk-reject it. The paper is worth the referee time.","headline":"A useful wrapper for symmetric LLM tasks with a solid debiasing core, but the exposure-transfer claim needs a direct test before the headline numbers are trusted.","tokens_in":718,"tokens_out":770,"would_cite":true,"duration_ms":134960,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reranking the input of a symmetric task—putting query-relevant elements where a model's attention lingers—can bring an LLM's accuracy to within about 1% of the optimal ordering.","keywords":["LLM input reranking","symmetric tasks","position exposure","relevance estimation","bipartite debiasing","Sinkhorn scaling","aggregate queries","long context"],"falsifier":"Run the exposure-discovery phase on a graph-degree task itself and compare the measured per-position miss rates to the profile learned from token counting; if the two differ, apply the claimed reranking and show its error is no better than random ordering, which would directly falsify the transfer claim.","tokens_in":19580,"feed_emoji":"🎯","tokens_out":6601,"duration_ms":64017,"temperature":0.7,"pith_summary":"Large language models forget parts of long inputs, and the forgetting is not uniform: items in some positions are much more likely to be missed than items in others. For symmetric tasks—queries about an unordered bag of elements, where any ordering is logically valid—this position bias can be exploited instead of suffered. The paper introduces input reranking: learn a position-by-position exposure profile for the target LLM, estimate how relevant each input element is to the query using a small helper model, then reorder the input so relevant elements sit in high-exposure positions. Experiments on graph-degree and database-aggregation tasks report that this wrapper improves accuracy by up to 99% of the gap between random ordering and an optimal ordering. If the claim holds, accuracy gains are available without retraining, new data, or changing the prompt content.","feed_headline":"Reranking inputs lifts LLM accuracy to 99 percent of optimal","feed_subtitle":"Placing query-relevant items where a model looks hardest nearly matches the best possible order on graph and database tasks.","key_machinery":"The exposure function $X_L(i)$ is the central object: it is the probability-like score that the target LLM $L$ will not miss an item at rank position $i$. The paper estimates it from random permutations of a toy token-counting task by solving $R^\\top X = \\vec{\\epsilon}$ in the least-squares sense, where $R$ records which tokens were relevant in each permutation and $\\vec{\\epsilon}$ holds the inverse of the observed output error (Eq. 10); a $t$-distribution confidence interval controls the number of permutations needed. For query-time relevance, the paper builds a bipartite evaluation graph whose left nodes are input elements and right nodes are helper-LLM evaluations of shuffled partitions, then alternates updates of element scores and evaluation-bias coefficients exactly like Sinkhorn scaling, which converges to a doubly stochastic limit and removes per-evaluation over- or under-scoring bias. Combining the two through Eq. 2 yields the reranking: relevant elements are moved to the positions the model actually sees.","core_discovery":"LLMs systematically overlook certain positions in long prompts, and the paper's central claim is that this position-dependent exposure can be measured and inverted into a better input order. The model treats the expected utility of a ranking as the position-weighted sum of element relevance, $E[utility(\\pi|q)] = \\sum_i X_L(i) \\cdot Rel_q(e_{\\pi(i)})$, where $X_L(i)$ is the exposure of position $i$ and $Rel_q$ is relevance to the query. Exposure is learned offline by fitting a least-squares system on a synthetic token-counting task whose per-token relevance is known, and query-time relevance is estimated by asking a helper LLM for coarse scores on shuffled partitions and debiasing those scores through iterative row/column scaling on a bipartite evaluation graph. In experiments on graph-degree and database aggregation tasks, the resulting rerankings reach roughly 97–99% of the optimal ranking utility in most settings and cut normalized output error nearly to zero, with the best helper models. The paper's claim is that reranking alone—no model change, no prompt-content change—recovers most of the accuracy that long-input forgetting loses.","pith_inferences":["The exposure-transfer assumption is the part most worth stress-testing: if per-position miss rates are measured directly on a graph or database task and they disagree with the token-counting profile, the reranking could be no better than random; a content-conditioned exposure model would be the natural fix.","The same machinery could be applied to retrieval-augmented generation, where the order of retrieved passages is controllable; the exposure profile would need to be re-estimated for that context structure.","A direct testable extension is to compare Eq. 9's linear model against a content-aware or interaction-aware error model on tasks where relevant items cluster semantically, since the linear model assumes position alone drives forgetting.","The bipartite debiasing step could be reused as a standalone bias-correction tool for any LLM-based scoring pipeline, including preference and rubric-based evaluation."],"forward_implications":["The same black-box wrapper applies to any symmetric task—edge lists, table rows, unordered collections—without task-specific prompt engineering.","Exposure profiles are model-specific, so the preprocessing must be repeated per target LLM; the paper observes that one evaluated model remembers the beginning best while another remembers the middle.","Query-time relevance estimation costs only $\\sigma m$ helper-LLM calls, so the overhead is small relative to the accuracy gain.","The bipartite debiasing procedure generalizes beyond reranking: any pointwise scoring by an LLM can be debiased the same way.","When reranking is used, normalized output error on several graph and database tasks lands close to the optimum, not just better than random."],"supporting_citations":[{"why":"Documents the 'lost in the middle' position-dependent retention pattern that motivates exposure discovery.","marker":"[25]"},{"why":"Supplies the exposure-weighted utility formula used to score candidate rankings.","marker":"[32]"},{"why":"Provides the Sinkhorn convergence result underlying the bipartite debiasing iterations.","marker":"[33]"},{"why":"Gives the nonnegative-matrix convergence condition used in the proof of Theorem 1.","marker":"[34]"},{"why":"Supplies the peer-review analogy for treating helper-LLM evaluations as biased reviewers.","marker":"[26]"},{"why":"Generates the Erdős-Rényi graphs used in the synthetic graph-degree evaluation.","marker":"[11]"},{"why":"Provides the IMDB movies table used for database aggregation queries.","marker":"[27]"},{"why":"Provides the OULAD student-enrollment table used for database aggregation queries.","marker":"[20]"},{"why":"Provides the Adult income table used for the hardest database aggregation queries.","marker":"[1]"}],"fun_headline_variants":["Input reranking lifts LLM accuracy to 99% of optimal","Rerank inputs to fix LLM blind spots in symmetric tasks","Place key items where LLMs look hardest to boost accuracy","Position-aware reranking recovers near-optimal LLM performance","Reranking inputs nearly eliminates LLM errors on bag queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the belief that the position profile learned on a synthetic single-token counting task is a fixed property of the model that carries over unchanged to graph and database tasks, and that the error falls linearly with the exposure-weighted sum of relevance.","fun_headline_variants_meta":{"raw":{"variants":["Input reranking lifts LLM accuracy to 99% of optimal","Rerank inputs to fix LLM blind spots in symmetric tasks","Place key items where LLMs look hardest to boost accuracy","Position-aware reranking recovers near-optimal LLM performance","Reranking inputs nearly eliminates LLM errors on bag queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1558,"prompt_tokens":1067,"completion_tokens":491,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":683,"tokens_out":491,"duration_ms":5078,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:14:32.221871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exposure-discovery phase on a graph-degree task itself and compare the measured per-position miss rates to the profile learned from token counting; if the two differ, apply the claimed reranking and show its error is no better than random ordering, which would directly falsify the transfer claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the exposure-weighted utility formula used to score candidate rankings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Sinkhorn convergence result underlying the bipartite debiasing iterations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the nonnegative-matrix convergence condition used in the proof of Theorem 1."},{"cited_title":"Integrating Rankings into Quantized Scores in Peer Review","cited_arxiv_id":"2204.03505","evidence_quote":"Supplies the peer-review analogy for treating helper-LLM evaluations as biased reviewers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Generates the Erdős-Rényi graphs used in the synthetic graph-degree evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the IMDB movies table used for database aggregation queries."}],"review_version":1}