{"id":"115ed357-2cf0-42ce-a28a-7401f3a655ed","arxiv_id":"1908.02484","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ESAC distributes RANSAC pose hypotheses among mixture-of-experts networks according to a gating distribution, improving camera re-localization accuracy in large, ambiguous environments.","lead":"This paper presents Expert Sample Consensus (ESAC), a way to train a team of specialist neural networks for camera location estimation, where a gating network splits the work among specialists instead of picking just one. It shows that this approach improves accuracy on large indoor datasets and makes progress on outdoor scenes where one big network struggles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ESAC's robustness to gating errors is asserted rather than measured: the method only recovers when the correct expert receives a nonzero share of the N hypotheses, and the paper never reports gating top-k recall or zero-allocation probability for failures.","rationale":"The reader's verdict is CONDITIONAL and identifies the gating distribution and the soft inlier score as the weakest assumptions. I agree with that general direction, but the single most load-bearing part is even narrower: the method can only recover if the correct expert receives at least one hypothesis, and the paper never measures how often this happens. This is the point where the central claim is least secure. The comparison to Expert Selection is well controlled (same pre-trained experts, same gating initialization, only the hypothesis-allocation/selection rule differs), so the per-scene pretraining and compute confounds are not the main issue. The lack of code and error bars is a real reproducibility concern, but it is not a specific challenge to the mechanism. The Oracle Gating comparison is informative: it shows the total loss from not knowing the scene ID is only 0.9 percentage points, which suggests the gating network, after ESAC training, is already very good at keeping the correct expert in play. However, it does not tell us whether the remaining failures are due to gating misses or due to consensus selecting the wrong hypothesis even when the correct expert had hypotheses. The proposed check directly measures the precondition in the paper's own claim. It would not invalidate the method if the decomposition shows gating misses are rare; it would simply define the regime in which ESAC's robustness holds. If the decomposition shows that zero-allocation risk is substantial, then the headline comparison to Expert Selection would need to be nuanced, because ESAC's advantage would be largely a higher top-k recall of the jointly trained gating network rather than the consensus rule itself. Either way, the verdict remains CONDITIONAL: the paper needs this analysis, or released code and seeds, before the central claim can be fully assessed. I therefore recommend no change to the reader's verdict.","tokens_in":16200,"tokens_out":17347,"duration_ms":161587,"concrete_test":"Retrain or reuse the 19Scenes ESAC model and, for every test image, record: (i) the gating probability of the ground-truth expert, g_c; (ii) the probability of zero allocation (1-g_c)^N with N=256; (iii) whether the correct expert is among the evaluated set and among the top-k gating experts for k=1,...,6; and (iv) the final pose error. Then tabulate ESAC failures by these quantities and decompose the 1.1% gap to Oracle Gating into (a) images where g_c was so low that zero allocation was likely and (b) images where the correct expert received hypotheses but the consensus score still selected a wrong pose. If (a) dominates, the robustness claim must be restated as conditional on gating top-k recall; if (b) dominates, the concern shifts to cross-expert score calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ESAC's central advantage rests on the claim that, as long as the gating network assigns some probability to the correct expert, distributing hypotheses and selecting by consensus can recover from gating mistakes. In the formulation of Eq. 10, hypotheses are allocated by a multinomial draw; an expert with gating probability p receives zero of the N=256 hypotheses with probability (1-p)^N. For p=0.01 this is 7.6%; for p=0.005 it is 27.7%. The paper reports 88.1% re-localization accuracy and 98.7% scene-classification accuracy on 19Scenes, but it never reports the distribution of gating probabilities assigned to the ground-truth expert, the top-k recall of the gating network (k=1,...,6), or the fraction of failures caused by the correct expert receiving zero hypotheses. Without this, the robustness claim is unfalsified: ESAC may be operating in a regime where the gating network already places the correct expert in the evaluated set with near-certainty, and the consensus mechanism may contribute far less than the comparison to the 53.0% top-1 gating accuracy suggests. The Oracle Gating result (89.0% vs 88.1%) shows that the total penalty for unknown scene ID is only 0.9%, but it does not decompose that penalty into gating misses versus score-comparison errors when the correct expert did receive hypotheses. The paper's own statement 'as long as the gating network attributes some probability to the correct expert' is therefore the load-bearing premise, and it is left unmeasured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Expert Sample Consensus (ESAC), which combines a Mixture-of-Experts architecture with Differentiable Sample Consensus (DSAC) for robust model fitting. Rather than letting a gating network select a single expert or averaging expert outputs, ESAC samples a multinomial split of a fixed hypothesis budget across experts and then selects the final hypothesis by sample consensus across all experts. The full ensemble, including the gating network, is trained end-to-end by minimizing the expected task loss with a K=1 gradient estimator. The method is evaluated on a synthetic line/circle toy problem, on indoor camera re-localization using 7Scenes, 12Scenes and their union 19Scenes, and on outdoor Dubrovnik and Aachen. On 19Scenes, ESAC re-localizes 88.1% of test images compared to 53.3% for DSAC++ and 47.5% for hard expert selection. A limitation paragraph in the appendix discusses remaining gaps to classical feature-based localization outdoors.","tokens_in":16572,"tokens_out":8406,"duration_ms":81076,"significance":"The paper makes a simple but effective conceptual contribution: distributing hypotheses according to gating probabilities while performing final selection by geometric consensus. This is a natural synthesis of MoE and DSAC, and the experiments provide strong evidence for its practical value. The oracle-gating and uniform-gating ablations in Fig. 8 cleanly bracket the performance of the gating network, and the end-to-end training ablation in Fig. 7 shows the benefit of joint optimization. The authors also honestly discuss the limitations of scene-coordinate regression on SfM-based outdoor ground truth. The main threat to the paper's conclusions is empirical: all reported numbers are single runs without error bars, and the claimed robustness to gating errors is not directly measured.","major_comments":[{"comment":"All accuracy numbers in the main comparison (Fig. 6) and in the speed/accuracy trade-off (Fig. 8) appear to be single-run point estimates; no error bars, confidence intervals, or significance tests are reported. This is particularly relevant for the toy experiment in Fig. 4, where the improvement of ESAC over Expert Selection is only +3.9%, and for the 19Scenes comparison where the margin is large but still subject to training stochasticity. Please report mean and standard deviation over multiple training runs, or at least a paired significance test, for the key comparisons (ESAC vs DSAC++ vs Expert Selection on 19Scenes, and ESAC vs Expert Selection on the toy problem).","section":"5.2, Fig. 6 and Fig. 8"},{"comment":"The paper's central robustness argument is that 'as long as the gating network attributes some probability to the correct expert' (Introduction), ESAC can recover from gating errors by giving that expert a nonzero share of the N hypotheses and selecting by consensus. This premise is never directly measured: the paper does not report the distribution of gating probabilities assigned to the ground-truth expert, the top-k recall of the gating network, or the frequency with which the correct expert receives zero hypotheses. For N=256, an expert with gating probability p=0.01 is excluded from the hypothesis pool with probability 7.6%, and with p=0.005 with probability 27.7%, so the operating regime matters. The comparison with Oracle Gating (89.0% vs 88.1%) shows that the total penalty for not knowing the scene ID is small, but it does not decompose that penalty into gating misses versus consensus-score errors when the correct expert did receive hypotheses. Adding this failure analysis would directly validate the claimed mechanism.","section":"3.3, Eqs. (10)-(11); 5.2, Fig. 8"}],"minor_comments":[{"comment":"The sentence 'It looses 3.5% accuracy' contains a typo: 'looses' should be 'loses'.","section":"5.2"},{"comment":"Please define how 'Classification Accuracy' is computed for ESAC in Fig. 6b; it appears to be the scene label of the expert that produced the selected hypothesis, but the text should state this explicitly.","section":"Fig. 6b"},{"comment":"In Eq. (16), the similarity measure has a slightly confusing arrangement of the exponent; please check that the formula as printed matches the intended Gaussian (e.g., whether sigma_e should be squared).","section":"Appendix B, Eq. (16)"},{"comment":"The phrase 'we me minimize the KL-divergence' contains a typo; it should read 'we minimize'.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound and the central contribution is likely publishable. The revisions requested are local and should be straightforward to address: adding error bars and a gating failure analysis. I recommend major revision rather than rejection because the missing experiments are feasible and do not require changing the method. One point to watch: the definition of classification accuracy for ESAC should be clarified, as it is currently ambiguous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper does something genuinely new: it merges DSAC with mixture-of-experts by distributing the N pose hypotheses across experts according to a gating distribution, then selecting via sample consensus. The joint training objective (Eq. 12) and the gradient estimator (Eq. 13) are clearly derived, and the experiments on 19Scenes show a large margin over DSAC++ and naive expert selection. That is real. The toy problem and the confusion matrices support the idea that ESAC hedges against gating mistakes better than hard selection.\n\nThe main soft spot is exactly what the stress-test note flags: the robustness claim pivots on the gating network assigning nonzero probability to the correct expert, but the paper never reports the distribution of gating probabilities for the correct expert, top-k recall, or the fraction of failures where zero hypotheses went to the right expert. The gap between Oracle Gating (89.0%) and ESAC (88.1%) suggests the consensus selection is doing most of the work once the right expert gets some hypotheses, but without that decomposition the sensitivity of the method to gating misses is under-measured. That said, the limited max-expert results in Fig. 8 are indirect evidence that the correct expert often sits near the top of the gating ranking, so the concern is real but not damning.\n\nOther soft spots are minor: the experiments appear to be single runs without variance, code is promised but not shipped, and the outdoor results, while improved, still trail classical feature-based methods. The authors candidly attribute this to SfM ground-truth outliers, which is a fair caveat.\n\nOn balance, this is a serious, well-written paper. The math is coherent, the comparisons are not circular, and the empirical claims are strong enough to justify a careful referee process. It deserves peer review, not desk rejection.","headline":"A coherent and well-motivated ensembling method for DSAC, with solid experiments; the main caveat is an unmeasured sensitivity to gating misses.","tokens_in":17081,"tokens_out":1651,"would_cite":true,"duration_ms":16809,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mixture-of-experts pose estimator that splits its hypothesis budget by gating and picks the winner by sample consensus reaches 88.1% re-localization on 19 combined rooms.","keywords":["Expert Sample Consensus","Mixture of Experts","camera re-localization","differentiable RANSAC","scene coordinate regression","sample consensus","end-to-end training","6D pose estimation"],"falsifier":"On a held-out subset of 19Scenes, force the gating probabilities so that the true scene's expert receives zero hypotheses on images the gating network currently gets wrong. If ESAC's accuracy does not fall below the reported 88.1% by a substantial margin, the paper's explanation that consensus rescues gating errors is not what is driving the result; if instead the accuracy collapses, the claim is supported.","tokens_in":15987,"feed_emoji":"📷","tokens_out":8475,"duration_ms":81219,"temperature":0.7,"pith_summary":"The paper tries to establish that, in a mixture-of-experts system for fitting parametric models, the gating network should neither choose a single expert nor average expert outputs. Instead, it should convert gating probabilities into a multinomial split of a fixed hypothesis budget, and the final estimate should be selected by sample consensus across all experts' hypotheses. This makes the ensemble forgiving of gating mistakes while retaining specialization, and it can be trained jointly and end-to-end. Applied to camera re-localization, the method reaches 88.1% correctly re-localized images on the combined 19Scenes environment, against 53.3% for the single-network baseline and 47.5% for hard expert selection.","feed_headline":"Consensus across expert pose guesses re-localizes 88% of images","feed_subtitle":"A gating network only divides a hypothesis budget among experts; geometric consistency picks the final pose.","key_machinery":"The central mechanism is the multinomial hypothesis split combined with a cross-expert softmax selection. Given M expert networks, each predicting scene coordinates, and a gating distribution g(e, I; w), ESAC draws a split H = (n_1, ..., n_M) with sum n_e = N from a multinomial distribution, gives expert e a budget of n_e pose hypotheses, and then selects the final pose by softmax over the soft inlier scores s(h_{e,j}, Y_e) of all N hypotheses. This makes the gating network's role one of resource allocation: as long as the correct expert receives a nonzero share, a good hypothesis can still win on geometric consistency. Training minimizes the expected task loss over both the split and the selection, with gradients approximated by sampling, including the log-derivative term for the split probability.","core_discovery":"Expert Sample Consensus (ESAC) claims that the correct way to combine a mixture of experts with differentiable RANSAC is not to let the gating network choose a single expert, nor to average expert outputs, but to convert gating probabilities into a multinomial split of a fixed hypothesis budget and then to select the final pose by sample consensus across all experts' hypotheses. The gating network therefore only decides how many of the N pose hypotheses each expert may propose; the geometric consistency score decides which hypothesis wins. On the combined 19-room indoor benchmark (7Scenes plus 12Scenes), ESAC re-localizes 88.1% of test images within 5 cm and 5 degrees, compared with 53.3% for the single-network DSAC++ baseline and 47.5% for hard expert selection, and it nearly matches an oracle that is told the correct scene (89.0%). The authors also report large gains over DSAC++ on outdoor datasets Dubrovnik and Aachen Day, while noting that sparse feature-based localization remains more accurate there.","pith_inferences":["The same budget-splitting idea should apply to other robust-fitting tasks with a discrete gate, such as object pose estimation or fundamental-matrix estimation, where the gating signal is weak or ambiguous.","Since only a nonzero share to the correct expert is needed, the gating network can be trained with substantially weaker supervision than a hard classifier; one testable consequence is that ESAC should tolerate noisier scene labels than expert selection.","The outdoor results suggest the remaining gap to feature-based localization is driven less by the ensemble design and more by the quality of the structure-from-motion ground truth used for training; filtering outlier 3D points before training is a natural follow-up.","The toy experiment's high model-type classification accuracy hints that ESAC could serve as a differentiable model-selection procedure, choosing not only parameters but the family of model, by letting experts compete through sample consensus."],"forward_implications":["On the combined 19-room environment, distributing hypotheses by gating reaches 88.1% correct re-localization versus 53.3% for a single network and 47.5% for hard expert selection, with the gap growing as the environment becomes larger.","Training the ensemble end-to-end with the ESAC objective matters most for precision: at a stricter 2 cm / 2 degree threshold, joint end-to-end training improves accuracy by more than 10 percentage points over the initialization.","Because experts with zero assigned hypotheses do not need to run, ESAC is faster than the monolithic baseline (555 ms versus 940 ms per image on 19Scenes) and can cap the number of active experts, gaining 19.7 percentage points over hard selection when only the top two experts are used.","The same formulation transfers to large outdoor scenes by dividing the environment into clusters and training an expert per cluster; on the Aachen Day benchmark ESAC raises correct localization from 0.4% to 42.6% at the strictest threshold as the number of experts grows, while still trailing sparse feature-based methods."],"supporting_citations":[{"why":"Provides DSAC, the differentiable hypothesize-and-verify estimator that ESAC extends to an ensemble.","marker":"[6]"},{"why":"Supplies the DSAC++ pipeline, pose refinement, and the single-network baseline ESAC is compared against.","marker":"[8]"},{"why":"Introduces Mixture of Experts and the gating formulation that ESAC reworks into hypothesis-budget allocation.","marker":"[20]"},{"why":"Defines RANSAC, the sample-consensus estimation paradigm underlying hypothesis scoring and selection.","marker":"[14]"},{"why":"Introduces scene coordinate regression and the 7Scenes dataset used for training and evaluation.","marker":"[44]"},{"why":"Provides the 12Scenes dataset that, combined with 7Scenes, forms the 19Scenes environment.","marker":"[48]"},{"why":"ActiveSearch is the sparse feature-based baseline that defines the performance gap on outdoor datasets.","marker":"[38]"},{"why":"Provides the Dubrovnik outdoor dataset used in the large-scale re-localization experiments.","marker":"[27]"},{"why":"Provides the Aachen Day outdoor dataset, including its evaluation protocol.","marker":"[39]"}],"fun_headline_variants":["Gating splits hypothesis budget; consensus picks final pose: 88% re-localized","No hard expert choice; consensus decides pose, 88% re-localized","Hypothesis budget split by gating, pose chosen by consensus: 88%","ESAC: experts propose, consensus disposes, 88% re-localized"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gating network gives the correct expert at least some of the hypothesis budget on every test image, and that the soft inlier score is a fair, comparable measure of hypothesis quality across different experts.","fun_headline_variants_meta":{"raw":{"variants":["Gating splits hypothesis budget; consensus picks final pose: 88% re-localized","No hard expert choice; consensus decides pose, 88% re-localized","Hypothesis budget split by gating, pose chosen by consensus: 88%","ESAC: experts propose, consensus disposes, 88% re-localized"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2263,"prompt_tokens":997,"completion_tokens":1266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1179}},"tokens_in":613,"tokens_out":1266,"duration_ms":10178,"temperature":1.0,"reasoning_tokens":1179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:42:02.494150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out subset of 19Scenes, force the gating probabilities so that the true scene's expert receives zero hypotheses on images the gating network currently gets wrong. If ESAC's accuracy does not fall below the reported 88.1% by a substantial margin, the paper's explanation that consensus rescues gating errors is not what is driving the result; if instead the accuracy collapses, the claim is supported.","supporting_citations":[{"cited_title":"DSAC-Differentiable RANSAC for camera localiza- tion","cited_arxiv_id":null,"evidence_quote":"Provides DSAC, the differentiable hypothesize-and-verify estimator that ESAC extends to an ensemble."},{"cited_title":"Learning less is more- 6D camera localization via 3D surface regression","cited_arxiv_id":null,"evidence_quote":"Supplies the DSAC++ pipeline, pose refinement, and the single-network baseline ESAC is compared against."},{"cited_title":"Jacobs, Michael I","cited_arxiv_id":null,"evidence_quote":"Introduces Mixture of Experts and the gating formulation that ESAC reworks into hypothesis-budget allocation."},{"cited_title":"Fischler and Robert C","cited_arxiv_id":null,"evidence_quote":"Defines RANSAC, the sample-consensus estimation paradigm underlying hypothesis scoring and selection."},{"cited_title":"Scene co- ordinate regression forests for camera relocalization in RGB- D images","cited_arxiv_id":null,"evidence_quote":"Introduces scene coordinate regression and the 7Scenes dataset used for training and evaluation."},{"cited_title":"Learn- ing to navigate the energy landscape","cited_arxiv_id":null,"evidence_quote":"Provides the 12Scenes dataset that, combined with 7Scenes, forms the 19Scenes environment."},{"cited_title":"Efficient & effective prioritized matching for large-scale image-based localization","cited_arxiv_id":null,"evidence_quote":"ActiveSearch is the sparse feature-based baseline that defines the performance gap on outdoor datasets."},{"cited_title":"Huttenlocher","cited_arxiv_id":null,"evidence_quote":"Provides the Dubrovnik outdoor dataset used in the large-scale re-localization experiments."},{"cited_title":"Benchmarking 6DoF outdoor visual localiza- tion in changing conditions","cited_arxiv_id":null,"evidence_quote":"Provides the Aachen Day outdoor dataset, including its evaluation protocol."}],"review_version":1}