{"id":"d4f36681-37f6-4780-9ad1-807ba87befe9","arxiv_id":"2501.15219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A DQN-based candidate selection and a competitive correction block improve MT ensembling quality while reducing inference cost on English-Hindi and Hindi-English tasks.","lead":"SmartGen uses a deep Q-network to pick a small set of translation models to ensemble, cutting inference cost while keeping translation quality. A second stage, SmartGen++, corrects weak candidates using a reward model and an LLM before fusing them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DQN selects candidates by independent top-K Q-values, but its stated goal is to select optimal groups; this mismatch is untested and could invalidate the central claim.","rationale":"The paper's central contribution is SmartGen, an RL-based candidate selection strategy that claims to 'identify optimal groups to pass to the fusion block.' The concrete mechanism, however, computes Q(s,a) for each individual candidate and takes the top K. This design is only justified if the optimal group is the set of K individually best candidates, i.e., if group value is additively separable. The paper's own motivation in Figure 2 demonstrates that the optimal triplet changes per sentence, but it does not distinguish between per-sentence variation in individual candidate quality and genuine combinatorial interactions; both would produce such variation. Without a direct comparison against the brute-force oracle, the core architectural assumption remains untested. I agree with the reader's weakest_assumption. I considered other issues: the Table 4 internal inconsistency (LLaMA-3-8B's 28.85 is below the SmartGen baseline of 28.92 despite the text claiming all enhancers surpass it), missing error bars, and private training data for the reward model. These are real reporting and reproducibility concerns, but they are secondary to the conceptual gap in the DQN's action space. If the oracle comparison shows SmartGen's triplets match the oracle closely, the concern is resolved and the conditional verdict stands. If not, the method's central claim is weakened and the paper would need to either adopt a set-valued action space or substantially temper its 'optimal group' language. Thus I recommend keeping the CONDITIONAL verdict, with the oracle comparison as the explicit acceptance condition.","tokens_in":13793,"tokens_out":4502,"duration_ms":40985,"concrete_test":"Using the same test set and fusion block as in Figure 2, compute the brute-force optimal triplet for each sentence. Run SmartGen's DQN to select its K=3 candidates. Report (i) the fraction of sentences where SmartGen's triplet exactly matches the brute-force optimal triplet, (ii) the mean BLEU gap between SmartGen's triplet and the oracle triplet, and (iii) SmartGen's average BLEU against a fixed global top-3 baseline and against random selection. If exact-match rate is low and the BLEU gap is non-trivial, the independent top-K assumption is the cause and the DQN must be reformulated as a set-valued selector (e.g., combinatorial action space or sequential selection with already-chosen candidates encoded in the state) before the group-identification claim can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the action space as {1,...,L} (individual candidate models) and selects the group bT as arg TopK_a Q(s,a). The reward, however, is the BLEU of the final fusion output, which depends on the whole selected set. A single-action Q-function is not equipped to represent the value of a combination: the Q-learning update for action a uses a reward that depends on which other candidates are simultaneously selected, so the per-candidate Q-values are not well-defined as standalone quality scores. The paper's own motivation (Figure 2) shows that the brute-force optimal triplet varies across sentences and no fixed subset dominates, which is consistent with combinatorial interactions among candidates. If group quality is not additively separable, greedily taking the K highest individual Q-values can systematically miss the optimal group. The paper never compares SmartGen's selected group against the brute-force oracle it constructed for Figure 2, so the core claim of 'identifying optimal groups to pass to the fusion block' rests on an untested independence assumption. This is load-bearing: if the DQN is only ranking individual candidates, the method reduces to a reranker, and the 'group optimization' contribution is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SmartGen and SmartGen++, two methods for ensembling machine translation systems. SmartGen uses a Deep Q-Network (DQN) to select a small, fixed number of candidate translations from a pool of L models, which are then passed to a fusion block; the DQN is trained using the BLEU score of the final fused output as reward. SmartGen++ adds a Competitive Correction Block (CCB) that uses a learned reward model and an LLM to rewrite weak selected candidates before fusion. The authors evaluate on English-Hindi and Hindi-to-English translation across several benchmarks, reporting BLEU, chrF++, and COMET scores, and claim reduced inference cost compared to ranker-based baselines while achieving competitive or better quality.","tokens_in":14036,"tokens_out":4804,"duration_ms":38050,"significance":"If the claims are substantiated, the paper addresses a relevant practical problem: the high inference cost of ensembling multiple NMT models. The idea of training the selection module with feedback from the fusion block is sensible, and the correction block is an interesting addition. The paper also provides a useful comparison of several ranking methods for NMT ensembling. However, the central contributions are weakened by a mismatch between the stated goal of selecting optimal groups and the actual DQN action space, by a factual contradiction in the reported correction results, and by the lack of statistical validation for very small BLEU gains. These issues must be resolved before the claims of state-of-the-art performance can be accepted.","major_comments":[{"comment":"The DQN selects the candidate set bT by taking the top-K per-action Q-values, but the reward is the BLEU of the fusion output over the entire selected set, so the per-candidate Q-value is not a well-defined standalone quality score because the reward for selecting candidate a depends on which other candidates are simultaneously selected. The paper's motivation (Figure 2) shows that the brute-force optimal triplet varies across sentences and no fixed subset dominates, which is consistent with combinatorial interactions among candidates, yet no experiment compares SmartGen's selected groups against the brute-force oracle used in that figure. The claim of \"identifying optimal groups to pass to the fusion block\" therefore rests on an untested additivity assumption; if the DQN is in effect only ranking individual candidates, the group-optimization contribution is not demonstrated.","section":"Section 3.1 and Figure 2"},{"comment":"The sentence \"Our findings indicate that the CCB block generates alternative translations that surpass the current translation on BLEU score metrics\" is contradicted by Table 4 for the Hindi-to-English direction: SmartGen's average BLEU is reported as 42.73, while all four LLMs used as the enhancer G in the CCB produce 42.28–42.30, which is below the current translation. This undermines the claim that CCB improves candidate quality in both directions and needs to be corrected, either by reporting separate results per dataset or by revising the claim.","section":"Section 4.6 and Table 4"},{"comment":"Using the reward model as the judge in Table 5 is partially circular. The reward model is trained (Section 3.2, Eq. 1) with preferred responses that include the top-3 BLEU candidates, and SmartGen is trained with BLEU-based rewards; therefore the higher reward for SmartGen/SmartGen++ over LLM-Blender may reflect the RM's inherited BLEU preferences rather than genuine quality improvements. The paper should present reference-based metrics as the primary evidence and treat the reward comparison as an auxiliary, or show agreement with an independent judge such as GPT-4 scores.","section":"Section 4.6 and Table 5"},{"comment":"Tables 2 and 3 report a single run without error bars or significance tests, and many of the reported gains are tiny (e.g., SmartGen improves over LLM-Blender by 0.22 BLEU on average for English-to-Hindi and by 0.21 for Hindi-to-English). Given the noise typical of BLEU on test sets of a few thousand sentences, such small differences cannot be distinguished from chance. The paper should report variance across multiple runs or bootstrap confidence intervals, and avoid claims of \"SOTA performance\" on the basis of marginal average improvements.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The notation \"arg, T opKa, Q(s, a)\" in Section 3.1 appears malformed; it should presumably denote selecting the K actions with the highest Q-values.","section":"Section 3.1"},{"comment":"The reward values in Table 5 (8.59, 15.35, 15.48) are inconsistent with the statement in Section 3.1 that the reward is normalized between [0,1]; either the normalization is not applied in this table or the description is inaccurate.","section":"Table 5 vs. Section 3.1"},{"comment":"The phrase \"SmartGen takes almost 2.31 times more faster than any of the Rankers\" is ungrammatical; it should read \"is 2.31 times faster.\"","section":"Section 4.5"},{"comment":"The word \"colliqual\" appears to be a typo for \"colloquial.\"","section":"Section 4.6"},{"comment":"The figure caption does not indicate which test set or dataset the brute-force analysis is performed on, making the motivation difficult to reproduce.","section":"Figure 2"},{"comment":"Table 1 reports a BLEU of 85.83 for the reference repeated K times, but it is unclear which test subset this is computed on and how many sentences it contains; please specify the subset size.","section":"Section 2.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a direct factual contradiction in Table 4 regarding the Hindi-English CCB results, and the central claim of group optimization by DQN is not supported by the presented experiments. These are fixable with additional analysis and corrected claims, so I recommend major revision rather than rejection. Please also consider whether the journal's standards require significance testing for such small BLEU differences."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2501.15219 (SmartGen). The core idea—using a DQN to pick K out of L MT candidates per sentence, with the fusion block's BLEU as reward—is a fresh twist on an important problem, and the Competitive Correction Block is a genuinely interesting addition. But the paper's central claim about selecting 'optimal groups' is not supported by the method as described, and the evaluation has some clear holes.\n\nThe good: framing candidate selection as RL and training with the final fusion reward is sensible, and the inference-cost reduction is real if the DQN actually beats the rankers on quality per dollar. The motivation experiments (Figure 2, Table 1) make the case that group choice matters and that weak candidates hurt fusion. That is worth taking seriously.\n\nNow the soft spots. The stress-test concern is on target: the DQN's action space is individual candidate models, and it selects the top-K Q-values independently, but the reward is the BLEU of the fused output, which depends on the whole group. If group quality is not separable, top-K individual values may miss the best group. The paper never compares SmartGen's chosen group to the brute-force oracle from Figure 2, so the group-optimization claim is untested. At best, SmartGen might be a reranker, which reduces the novelty.\n\nThe evaluation also lacks error bars and significance tests; many gains are around 0.2 BLEU, and in some datasets SmartGen is worse than LLM-Blender. There is a direct inconsistency: the text says the CCB produces candidates that surpass the current translation, but Table 4 (Hindi-English) shows all corrected candidates scoring below SmartGen's 42.73. That needs attention. The reward model is partly trained on BLEU-preferred responses, so using it as a judge (Table 5) is circular. Also, only English-Hindi/Hindi-English, private data, no code or model release.\n\nAll that said, the paper deserves a serious referee. The problem is practical and the approach has legs, but the authors need to either justify the top-K independence assumption or change the formulation, fix the Table 4 reading, add statistical rigor, and release artifacts. I'd send it to review with an expectation of major revision.","headline":"SmartGen has a promising RL-based approach to MT ensembling, but the group-selection claim rests on an untested independence assumption and the evaluation has clear gaps.","tokens_in":14549,"tokens_out":3161,"would_cite":false,"duration_ms":26797,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that candidate selection, not fusion, is the bottleneck in neural machine translation ensembling, and that a Deep Q-Network trained on fusion-block rewards — plus a competitive correction step — produces better…","keywords":["machine translation ensembling","deep Q-network","candidate selection","reinforcement learning","fusion block","competitive correction","English-Hindi translation","neural machine translation"],"falsifier":"On a held-out test set, compute the brute-force optimal K-subset per sentence by fusing all C(L,K) combinations and compare it with the DQN's top-K choice: if the DQN rarely matches the oracle yet trails it by a large BLEU margin, the additive group assumption is the cause. A direct test is to train a DQN whose actions are entire K-subsets; if that variant does not beat top-K selection, group non-separability is not the binding constraint.","tokens_in":13564,"feed_emoji":"⚡","tokens_out":11445,"duration_ms":90467,"temperature":0.7,"pith_summary":"This paper claims the bottleneck in neural machine translation ensembling is candidate selection, not the fusion model itself, and that casting selection as a reinforcement learning problem fixes it. SmartGen uses a Deep Q-Network to pick, for each source sentence, a small fixed set of K candidate translations from a pool of L models, passing only those to an encoder-decoder fusion block, and trains the picker with the fusion block's BLEU score as reward so selection and fusion are optimized jointly. The follow-on SmartGen++ adds a Competitive Correction Block that scores the chosen candidates with a reward model and, when a weak one is detected, rewrites it with a large language model conditioned on the rejected candidates. On English-Hindi and Hindi-English benchmarks, the paper reports that SmartGen is about 2.31 times faster than ranker-based ensembling and that SmartGen++ beats the best ranker baseline by 4.48% BLEU. If the claims hold, ensembling can be made both cheaper and better by aligning the selection objective with the fusion block's output.","feed_headline":"RL selects the best K translations, making ensembling 2.3x faster","feed_subtitle":"Per-sentence DQN selection cuts ensemble cost 2.3x; a correction block pushes BLEU still higher.","key_machinery":"The load-bearing mechanism has three parts. First, a Deep Q-Network with a ResNet backbone maps each source sentence (state) to Q-values over the L candidate model indices (actions); the group passed to the fusion block is the top-K Q-values, and the training reward is the sacreBLEU of the fusion block's output, stored in an experience replay buffer. Second, a reward model rθ(x, y), trained with a modified preference loss over sets of four preferred and all rejected candidates, scores every candidate and reveals weak selected candidates by the margin between their rewards. Third, a correction block takes any candidate whose reward margin falls below a threshold τ, plus the rejected candidates and their scores, and asks an enhancer LLM G to produce a replacement translation; that replacement enters the fusion set. The paper's guiding identity is that a candidate group is only as good as what the fusion block does with it, so the selector's objective is the fusion block's output quality, not an independently learned ranking.","core_discovery":"The paper's central claim is that the usual 'score candidates, keep the top K, fuse' recipe is doubly flawed: the scorer is trained independently of the fusion block, and the fused output is dragged down by the worst candidate in the selected group. SmartGen redefines the task as an MDP in which the state is the source sentence, the actions are candidate model indices, and the reward is the sacreBLEU of the translation produced by the fusion block from the selected group; the DQN selects the group by taking the top-K Q-values. SmartGen++ adds a Competitive Correction Block whose reward model, trained on sets of preferred and rejected candidates, flags weak selected candidates by their reward margin, and an LLM then rewrites them using the rejected candidates as context. The experiments are meant to show that joint training improves quality over rankers that ignore the fusion block, that the correction step recovers quality lost to weak candidates, and that selecting only K models cuts the ensemble's inference time from Ω(L) to O(K).","pith_inferences":["If the paper's central claims hold, the top-K Q-value rule is the likely weak point, because the paper's own brute-force experiment shows the optimal triplet changes per sentence, implying group value is not separable into per-candidate values; a set-valued action space or beam search over K-subsets should close part of the gap to the oracle.","The reward model, trained on preferred and rejected candidate sets, could serve as a reference-free quality judge; the ablation compares reward scores to GPT-based scoring, but the paper stops short of proposing the reward model as a metric itself.","Because the Competitive Correction Block is described as decoupled from the DQN, a natural extension is to attach it to PairRanker or SimCLS; if it lifts those baselines as much as it lifts SmartGen, the correction step is doing the heavy lifting rather than the RL selector.","The paper admits K is fixed; extending the DQN to emit a per-sentence group size would exploit the observed sentence-to-sentence variation in optimal triplet composition, though the paper does not pursue variable K."],"forward_implications":["Ensembling cost drops from running all L candidate models to running only the K selected ones per sentence — an Ω(L) to O(K) reduction in model inference time.","Because the DQN's reward is the fusion output's sacreBLEU, the selector is trained to pick groups that fuse well rather than individually strong translations, closing the gap identified in the motivation.","The Competitive Correction Block decouples candidate improvement from selection: any select-and-fuse ensemble can bolt it on to repair weak candidates, which the paper notes is of independent interest.","Joint training and correction together produce the reported gains: SmartGen++ tops the BLEU, chrF++, and Comet tables on most English-Hindi and Hindi-English test sets, while the DQN is the fastest ranker in the comparison."],"supporting_citations":[{"why":"supplies the LLM-Blender baseline with PairRanker and the generative fusion block that SmartGen compares against and builds on; its K=3 choice is adopted.","marker":"(Jiang et al., 2023)"},{"why":"provides the Deep Q-Network algorithm and experience replay that implement the candidate selection block.","marker":"(Mnih et al., 2013)"},{"why":"defines sacreBLEU, the metric used as the DQN reward and the paper's primary evaluation.","marker":"(Post, 2018)"},{"why":"is the prior select-and-fuse fusion method where the selection block and fusion block are trained independently — the setup SmartGen claims is suboptimal.","marker":"(Hoang et al., 2024b)"},{"why":"contributes SimCLS, a contrastive ranking baseline whose selection objective ignores the fusion block and which SmartGen outperforms.","marker":"(Liu and Liu, 2021)"},{"why":"contributes SummaReranker, a reranking baseline used in the comparison that SmartGen must beat.","marker":"(Ravaut et al., 2023)"},{"why":"is the reward-modeling framework whose single-preference loss the CCB's reward model modifies to preferred/rejected candidate sets.","marker":"(Stiennon et al., 2022)"}],"fun_headline_variants":["RL picks top-K translations, cutting ensembling cost 2.3x","DQN selects K candidates, then an LLM fixes weak ones","SmartGen: RL-guided ensemble selection with competitive correction","Fast NMT ensembling: RL chooses K, correction boosts BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DQN selects the group by taking the K candidates with the highest individual Q-values, which assumes the value of a group equals the sum of its parts; if the fusion block's gain depends on combinatorial interactions between candidates, this top-K rule can miss the optimal group even when the Q-values are perfectly learned.","fun_headline_variants_meta":{"raw":{"variants":["RL picks top-K translations, cutting ensembling cost 2.3x","DQN selects K candidates, then an LLM fixes weak ones","SmartGen: RL-guided ensemble selection with competitive correction","Fast NMT ensembling: RL chooses K, correction boosts BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1501,"prompt_tokens":946,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":562,"tokens_out":555,"duration_ms":4833,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:29:10.494485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out test set, compute the brute-force optimal K-subset per sentence by fusing all C(L,K) combinations and compare it with the DQN's top-K choice: if the DQN rarely matches the oracle yet trails it by a large BLEU margin, the additive group assumption is the cause. A direct test is to train a DQN whose actions are entire K-subsets; if that variant does not beat top-K selection, group non-separability is not the binding constraint.","supporting_citations":[],"review_version":1}