{"id":"c2090292-0aab-456b-9481-733587f46d0f","arxiv_id":"2601.22211","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A spherical flow-matching policy over latent cost directions, mapped to feasible actions by a combinatorial solver with a vMF-smoothed value critic, beats prior combinatorial-RL baselines by 20.6% on four benchmark tasks.","lead":"LSFlow learns a stochastic policy for combinatorial action problems by sampling a direction on a sphere and letting an optimization solver convert that direction into a feasible action. It offers a way to combine expressive generative policies with hard feasibility guarantees for scheduling, routing, and network-based disease testing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The critic's vMF-smoothed fixed point Qκπ is the value of the perturbed policy, but the evaluated policy and the policy-improvement theorem use unperturbed Q(s,a⋆(s,c)); the gap is never bounded, so training may optimize a surrogate different from the reported objective.","rationale":"I agree with the reader's weakest assumption: the smoothed Bellman operator creates a value function for the vMF-perturbed policy, but the policy actually evaluated at test time appears to be the unperturbed center policy, and the policy-improvement theorem invokes the unperturbed Q(s,a⋆(s,c)). This is load-bearing because the actor's reweighting uses the critic's estimate of Qκπ, so if the gap between Qκπ and the true value of the evaluated policy is large, the algorithm optimizes a surrogate objective whose optimum may differ from the reported returns. The paper acknowledges only a 'bias-variance trade-off' (Appendix A) and never bounds this gap; Theorem 3.5 guarantees smoothness but not fidelity to the unperturbed value. This concern is concrete and addressable: on small instances, exact value iteration can quantify the gap, and a corrected derivation of Theorem H.1 with Qκπ would clarify the objective actually being optimized. The empirical claim of 20.6% improvement may still hold, but the theoretical support and the interpretation of the trained policy would need revision. Since this concern is the same one the reader identified and does not, on the available evidence, break the central idea, the conditional verdict should stand.","tokens_in":23962,"tokens_out":7205,"duration_ms":80570,"concrete_test":"On a small dynamic-scheduling instance (e.g., N=10, B=3, H=10), compute exact Qπ and Qκπ by value iteration for the trained center policy π and κ=28, and compare. Specifically: (1) evaluate the reported test-time policy (sample c∼π, execute a⋆(s,c)) and the perturbed policy (also sample c̃∼Kκ(·|c) before solving) in the environment; (2) for the same π, compute ||Qκπ−Qπ||∞ over sampled (s,c) pairs. If the returns differ materially, or the sup-norm gap is large, the actor is optimizing a surrogate different from the evaluated objective. To settle the theory, re-derive Theorem H.1 with Qκπ in place of Q(s,a⋆(s,c)) and exhibit the extra term E_{c̃∼Kκ}[Q(s,a⋆(s,c̃))−Q(s,a⋆(s,c))].","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central policy-update justification (Theorem H.1, Eq. 15) applies to Q(s,a⋆(s,c)), the true value of the solver action. The algorithm instead trains the critic with the vMF-smoothed target Eq. (8), whose fixed point Qκπ (Theorem 3.5) is the value of the perturbed policy πκ, not of the policy evaluated at test time. During data collection (Algorithm 1 lines 6–8, Sec. 3.4) the agent draws c∼π(·|s), perturbs c̃∼Kκ(·|c), and executes a⋆(s,c̃). At inference, Figure 1 shows the unperturbed path c→solver→action, and the paper never states that test-time evaluation applies the same Kκ perturbation. Hence Q̃(s,c)≈Qκπ(s,c)=E_{c̃∼Kκ(c)}Q^{πκ}(s,a⋆(s,c̃)), while the evaluated policy is π(a⋆(s,c)) with c∼π. The KL-regularized interpretation in Eq. (15) uses Q(s,a⋆(s,c)), not Qκπ(s,c); replacing one by the other changes the objective by a gap that Appendix A only calls a 'bias-variance trade-off' but never quantifies. Without a bound such as ||Qκπ(s,·)−Qπ(s,a⋆(s,·))||∞ ≤ f(κ, geometry of the normal fan), there is no guarantee that training maximizes the return of the evaluated policy. The empirical gains could therefore be driven by the critic's smoothing bias rather than by the expressiveness of the spherical flow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LSFlow, a policy class for RL with combinatorial action spaces. A spherical flow model samples a cost direction c; a combinatorial solver maps c to the feasible action a⋆(s,c), guaranteeing feasibility by construction. To avoid repeated solver calls, both actor and critic are trained in the latent cost space. The critic is trained with a vMF-smoothed Bellman operator intended to smooth solver-induced discontinuities, and the actor is updated by weighted spherical flow matching. The paper proves exact expressivity of solver-induced policies (Prop. 3.2), contraction and C∞ smoothing of the smoothed operator (Thm. 3.5), and equivalence of weighted flow matching to KL-regularized policy improvement (Thm. H.1). Experiments on four public benchmarks and an STI testing task report gains over SEQUOIA, SRL, and DQN-Sampling, with an average 20.6% improvement over SRL.","tokens_in":24334,"tokens_out":15355,"duration_ms":171404,"significance":"The framework is attractive: it combines generative stochastic policies with hard feasibility, and the latent-space critic is a sensible way to avoid solver-in-the-loop training. If the theoretical claims were correct, this would be a meaningful step beyond deterministic structured policies for combinatorial RL. The empirical comparison is against established external baselines, and the reported efficiency gain is substantial. Proposition 3.2 and Theorem 3.5 are, for the operator as defined, largely sound. However, the theoretical scaffolding currently has two load-bearing gaps: the smoothed operator's fixed point is not the value of the policy executed during data collection, and Theorem H.1's equivalence fails under a shared parameterization. These issues must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The operator Tπκ defined in Eq. (7) is not the Bellman operator of the policy executed during data collection. Algorithm 1 samples a center c∼π(·|s), perturbs it to c̃∼Kκ(·|c), executes a⋆(s,c̃), and stores (s,c,r,s′). At the next state the stored center is a fresh sample c′∼π(·|s′), not c̃′. The Bellman operator for this policy would be (TQ)(s,c)=E_{c̃∼Kκ(c), s′|c̃, c′∼π(s′)}[r(s,a⋆(s,c̃))+γ Q(s′, c′)]. Eq. (7) instead evaluates Q(s′, c̃′) with c̃′∼Kκ(c′). Its fixed point is the value of a 'doubly perturbed' policy whose next center is the perturbed direction, not the center stored in the buffer. Consequently, the target (8) bootstraps with the wrong argument, and the critic does not estimate the value of the executed policy. This is load-bearing because the actor weights (Algorithm 1, line 19) are built from this critic.","section":"Section 3.3, Eq. (7); Section 3.4, Eq. (8)"},{"comment":"The proof claims that because Z_k(s)>0 is independent of θ, multiplying each per-state objective by Z_k(s) does not change the minimizers. This is true only if minimization is performed separately for each s. For a shared parameter θ, the objective L_k(θ)=E_{s∼D}[Z_k(s) B_s(θ)] and the unweighted objective \\tilde L_k(θ)=E_{s∼D}[B_s(θ)] generally have different minimizers. For example, with two states, Z=(100,1) and B_s(θ)=(θ±1)^2, the weighted minimizer is near 0.98 while the unweighted minimizer is 0. Thus Eq. (4) is not equivalent to unweighted flow matching with c1∼π_{k+1}, and the KL-regularized interpretation of the concrete algorithm is not established. The theorem would need to either restrict to per-state policies or redefine the outer sampling to reweight states by 1/Z_k(s).","section":"Appendix H, Theorem H.1(i), Eqs. (13)–(14)"},{"comment":"Even if Eq. (7) were corrected, the policy-improvement justification in Appendix H concerns the unperturbed value Q(s,a⋆(s,c)), whereas the algorithm trains the critic on the smoothed operator and uses the resulting \\tilde Qφ(s,c) as the reweighting score. The paper never bounds ∥Qκπ(s,c) − Q^{π}(s,a⋆(s,c))∥_∞, nor does it state whether test-time evaluation applies the vMF perturbation (Figure 1 shows an unperturbed path). Without such a bound or an explicit test-time protocol, the reported gains may be attributable to the smoothing bias rather than to the flow policy's expressiveness. Appendix A only calls this a bias-variance trade-off; Section 3.3/3.4 needs a formal or at least a clearly stated consistency argument.","section":"Section 3.2/3.4 vs Appendix H, Eq. (15); Appendix A"}],"minor_comments":[{"comment":"The Random/Greedy baselines for Dynamic Assignment are described using 'simple-cycle sampling' from Dynamic Routing, which is not meaningful for the assignment environment. Please correct the description.","section":"Appendix I.2, Dynamic Assignment"},{"comment":"The inference path in Figure 1 omits the vMF perturbation used during training. Please state explicitly whether test-time evaluation uses the same perturbation.","section":"Figure 1 and Section 3.4"},{"comment":"The number of random seeds is not reported. Please add seed counts and, if available, standard errors.","section":"Table 1"},{"comment":"The claim of being the first flow/diffusion policy for combinatorial RL should be reconciled with the cited concurrent work Ma et al. (2025b), whose title explicitly includes 'combinatorial action spaces'.","section":"Section 4 and References"},{"comment":"Minor typos: 'Transimitted' in the Appendix title; 'PSocessing' in the Ho et al. reference; 'desgnated' in Appendix I.1.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an appealing framework and credible empirical results, but the theory as written contains two fixable errors: the smoothed Bellman operator does not correspond to the executed policy, and Theorem H.1's equivalence fails under shared parameters. The authors should also clarify the test-time protocol and report seed counts. I recommend major revision rather than rejection, as the core idea is promising and the technical issues appear addressable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core idea is solid and genuinely new: learn a distribution over cost directions on the sphere, push each sample through a combinatorial solver, and get a stochastic policy over feasible actions with feasibility guaranteed by construction. Proposition 3.2 is the real contribution — every action is a vertex, hence an exposed point, so its strict-optimality cone has positive measure and any target action distribution can be induced exactly. That proof is correct. Theorem 3.5 (contraction plus C-infinity smoothing of the value in c) also checks out. The latent-space critic is a sensible way to avoid solver calls in the policy-improvement loop. This is the first flow-style generative policy for combinatorial RL I know of, and the idea deserves attention.\n\nThe soft spots are real but not fatal. The main one is the surrogate gap. The critic is trained with vMF-smoothed Bellman targets; its fixed point Q^pi_kappa is the value of the perturbed policy (sample c~pi, then c~K(·|c), execute a*(s,c~)). But inference uses the unperturbed path c -> a*(s,c), and the KL-regularized policy-improvement theorem (H.1) uses unperturbed Q(s,a*(s,c)). The paper never bounds the difference between Q^pi_kappa and the true Q^pi, and Appendix A only calls it a bias-variance trade-off. So the training objective is a surrogate whose fidelity is assumed, not shown. That is fixable, but the theory as written overclaims.\n\nThe empirical side is thinner. No code or data released, and Table 1 reports no seeds or significance tests, so the 20.6% average improvement over SRL is not statistically verified. SRL is a self-implementation in PyTorch, which is a risk if the original Julia implementation behaves differently. The STI graph construction retains the subgraph with the most positive nodes, which could skew results in a way that favors structure-exploiting methods. One minor point: I actually think Theorem H.1(i) is fine — Z_k(s) is positive and independent of theta, so the minimizer set is unchanged.\n\nThis paper deserves a serious referee. The core construction is novel and the theory is mostly sound; the gaps are addressable. Send it to review, and ask the authors for code, seeds with significance reporting, an honest discussion of the perturbed-vs-evaluated policy gap, and a check on the STI subgraph selection.","headline":"A genuinely new stochastic-policy construction for combinatorial RL with sound core theory, but the training objective optimizes a smoothed surrogate whose fidelity to the evaluated policy is unquantified, and the empirical claims lack code and significance tests.","tokens_in":24897,"tokens_out":2147,"would_cite":true,"duration_ms":26844,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A spherical flow over cost directions plus a feasibility-enforcing solver yields an expressive stochastic policy for combinatorial RL, outperforming the strongest prior baseline by 20.6% on average.","keywords":["combinatorial reinforcement learning","flow matching policies","spherical latent space","solver-induced stochastic policies","feasibility by construction","smoothed Bellman operator","von Mises-Fisher kernel","KL-regularized policy improvement"],"falsifier":"On a small combinatorial MDP with a known optimal action, shrink the spherical measure of that action's optimality region to near zero and apply moderate-to-strong smoothing (small kappa); if the smoothed fixed point's implied action ranking disagrees with the true Q ranking, or if the empirical gap between the critic Q~_phi(s,c) and the actual discounted return of executing a*(s,.) grows as kappa decreases, then the surrogate is unfaithful and the reported gains depend on the smoothing regime rather than the policy class. A cheaper observation: in the paper's own benchmark setup, compare the","tokens_in":23741,"feed_emoji":"🎯","tokens_out":10889,"duration_ms":109275,"temperature":0.7,"pith_summary":"Reinforcement learning with combinatorial actions — scheduling, routing, assignment, budgeted subset selection — has been stuck between value functions embedded into solvers and deterministic structured policies. This paper argues that the expressiveness of generative (flow) policies can be brought to these domains by splitting the problem: learn a stochastic distribution over cost directions on the unit sphere, and let a combinatorial solver convert each sampled direction into a feasible action. The sphere is the natural latent space because the solver's argmin depends only on a cost vector's direction, and spherical flow matching can represent rich multimodal distributions there; the paper proves both that this policy class can represent any stochastic policy over a finite feasible set and that a vMF-smoothed Bellman operator has a unique, infinitely smooth fixed point, stabilising value learning across the solver's discontinuous decision boundaries. Everything is trained in latent cost space — the critic, the reweighted flow objective, the smoothed targets — so the solver runs once per environment step rather than inside the learning loop. The reported payoff is an average 20.6% reward gain over the strongest baseline on four benchmark tasks and better detection efficiency on a real-world STI batch-testing task, suggesting generative stochastic policies are a workable route for combinatorial RL.","feed_headline":"Beats combinatorial-RL baselines by 20.6% on average","feed_subtitle":"Sampling cost directions and letting a solver guarantee feasibility brings generative policies to combinatorial RL.","key_machinery":"The load-bearing construction is the solver-induced two-stage policy: c maps to a*(s,c) = argmin_{a in A(s)} c^T a, with the cost vector restricted to the unit sphere, since positive scale invariance means only the direction matters. Feasibility is guaranteed by the solver, while all stochasticity lives in a spherical flow model pi_theta(c|s), trained by spherical flow matching — a generative model that transports a base distribution on the sphere along a projected ODE. The second mechanism is the von Mises-Fisher (vMF) smoothed Bellman operator, a bell-shaped on-sphere local average of reward and bootstrap terms; the paper proves it is a gamma-contraction whose unique fixed point is C^infin","core_discovery":"The paper's central claim is that the feasibility bottleneck in combinatorial RL can be decoupled from policy expressiveness: the policy never selects discrete actions directly. It samples a cost direction c on the unit sphere S^{m-1}; the solver mapping a*(s,c) = argmin_{a in A(s)} c^T a turns that direction into a valid structured action, so feasibility holds by construction (Lemma 3.1 justifies the spherical domain via positive scale invariance). Proposition 3.2 shows the induced policy class is exactly expressive: every feasible action is a vertex of the convex hull of the feasible set, hence occupies a nonempty open region of the sphere, and a mixture over these regions reproduces any t","pith_inferences":["The exact-expressivity proof (Proposition 3.2) is a representation guarantee, not a learnability guarantee: it shows some distribution over the sphere induces any target policy, but says nothing about whether a spherical flow can fit that distribution from finite samples. A natural stress test is whether the learned policy's support collapses onto a few solver regions on complex instances.","The surrogate gap noted in the paper's limitation section — smoothing introduces a bias-variance trade-off — suggests a concrete extension the authors leave open: a state-dependent or annealed concentration parameter kappa that starts wide (strong smoothing, stable targets) and narrows as training proceeds, which could recover unbiased value estimates without sacrificing stability.","The two-stage design pattern generalizes beyond linear objectives: any solver interface that maps a continuous parameter into a feasible action could inherit the same stochastic-policy machinery, with the spherical geometry replaced by whatever invariance group the solver's objective admits.","One implementation detail worth probing: the behavior policy executes a*(s,c~) for a perturbed direction c~ while the critic is trained on the stored center c, so rewards come from the perturbed distribution but critic inputs from the center distribution; the paper treats this as matching the smoothed operator, but the mismatch is not analyzed separately."],"forward_implications":["As far as the authors know, this is the first flow- or diffusion-based policy framework for RL with combinatorial action spaces; if correct, it transfers the expressiveness of generative policies to constrained discrete decision-making.","Because feasibility is delegated to an off-the-shelf solver, the same spherical policy machinery transfers across constraint types — routing, scheduling, assignment, budgeted selection — without re-engineering the policy's geometry for each domain.","The C^infinity fixed point of the smoothed Bellman operator means critic targets are provably smooth in the cost direction, so the policy-gradient noise induced by boundary-crossing solver decisions is mitigated rather than patched.","Training in latent cost space rather than action space cuts policy-update cost by roughly two orders of magnitude (about 6 minutes per update for the action-space critic versus under 1 second for the cost-space critic on the scheduling task), which is what makes the stochastic policy practical."],"fun_headline_variants":["Latent sphere flow policy beats combinatorial RL baselines by 20.6%","Solver-induced spherical flow policy: 20.6% better on combinatorial RL","Feasibility by construction: latent flow policy for combinatorial RL gains 20.6%","Spherical flow matching: solver guarantees feasibility, 20.6% better RL","20.6% gain: latent spherical flow policy with solver-induced feasibility"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the vMF-smoothed Bellman operator's unique fixed point — the value of the perturbed policy, averaged over random cost directions — faithfully approximates the value of the policy actually executed; the paper never bounds this surrogate gap, so if smoothing bias is large in a given domain, LSFlow provably optimizes a smoothed problem whose optimum can differ from the original.","fun_headline_variants_meta":{"raw":{"variants":["Latent sphere flow policy beats combinatorial RL baselines by 20.6%","Solver-induced spherical flow policy: 20.6% better on combinatorial RL","Feasibility by construction: latent flow policy for combinatorial RL gains 20.6%","Spherical flow matching: solver guarantees feasibility, 20.6% better RL","20.6% gain: latent spherical flow policy with solver-induced feasibility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3373,"prompt_tokens":728,"completion_tokens":2645,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2539}},"tokens_in":472,"tokens_out":2645,"duration_ms":19015,"temperature":1.0,"reasoning_tokens":2539,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:45:00.982307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small combinatorial MDP with a known optimal action, shrink the spherical measure of that action's optimality region to near zero and apply moderate-to-strong smoothing (small kappa); if the smoothed fixed point's implied action ranking disagrees with the true Q ranking, or if the empirical gap between the critic Q~_phi(s,c) and the actual discounted return of executing a*(s,.) grows as kappa decreases, then the surrogate is unfaithful and the reported gains depend on the smoothing regime rather than the policy class. A cheaper observation: in the paper's own benchmark setup, compare the","supporting_citations":[],"review_version":1}