{"id":"ae87acc4-105c-4167-b6d0-0b00fcd0d8cd","arxiv_id":"2411.19602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reinforcement-learned one-dimensional Ising spins show disorder, flocking, flipping, and oscillatory phases, with reversal time decaying exponentially in the exploration rate.","lead":"A physics team trained one-dimensional Ising spins with a reinforcement learning rule that rewards keeping neighbors, and mapped the resulting phases in a speed-versus-noise plane. The work shows that machine-learned flocking reproduces known directional-switching behavior and adds a newly described oscillatory, chaotic-looking phase.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (3) is not a Bellman update; if implemented literally, learned Q values alternate sign and the argmin policy can produce artificial flipping/oscillation. This ambiguity undermines the reproducibility of the entire phase diagram and reversal-time scaling.","rationale":"The reader's weakest assumption correctly identifies Eq. (3) as the most fragile load-bearing premise. The central claim is that a Q-learning rule with a cohesion-only cost produces four phases, including flipping and oscillatory phases with specific scaling and chaotic behavior. All of these results flow from the update rule; if the rule is misprinted, the learning dynamics are not Q-learning at all, and the observed phases may be numerical artifacts of an anti-persistent Q value rather than emergent collective behavior. The paper gives no independent support for the printed update: there is no code deposit, no data repository, and no comparison to a standard Bellman implementation. The Lyapunov-exponent analysis is also underdocumented, but it is secondary because the existence and scaling of the phases are claimed before chaos is invoked. I do not see an internal inconsistency that forces rejection; the correct response is a conditional acceptance requiring the authors to clarify or fix Eq. (3), provide code or detailed pseudocode, and supply the missing chaos-analysis parameters. This matches the reader's CONDITIONAL verdict, so no adjustment is needed.","tokens_in":14251,"tokens_out":4595,"duration_ms":39873,"concrete_test":"Obtain or reimplement the simulation and run the identical protocol twice, changing only the update rule in Eq. (3): (i) as printed, Q <- -(1-alpha)Q + alpha*C; (ii) standard Bellman, Q <- (1-alpha)Q + alpha*C. Compare the phase diagram in the (epsilon, v-bar) plane, the <T> vs epsilon curves, and the Lyapunov-exponent estimates for the oscillatory phase. If the flipping and oscillatory phases and the exponential reversal-time scaling survive in version (ii) with the same exponents within error bars, the minus sign is not load-bearing; if they change qualitatively or vanish, the central claims are artifacts of the non-standard update. Additionally, report the embedding parameters (delay, dimension, evolution time) and surrogate tests for the Wolf-algorithm Lyapunov exponents.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (3) in Section II is written as Q_i <- -Q_i(1-alpha) + alpha*C_i. The standard Q-learning update for a minimization problem with zero discount factor is Q_i <- (1-alpha)Q_i + alpha*C_i. The printed minus sign on the previous Q value has no standard justification and is not derived in the paper. If the simulations implement the printed rule, Q values for a given state-action pair flip sign on successive visits (roughly alpha, then -alpha(1-alpha), then alpha(1-alpha)^2, ...). Because exploitation in Eq. (4) selects argmin Q, the chosen action will oscillate on a timescale set by visit frequency, generating artificial switching that could masquerade as the flipping and oscillatory phases. If instead the minus sign is a typo and the code uses the standard update, the results depend on an undocumented correction and cannot be reproduced from the text alone. The paper provides no code or data, and the phase diagram is drawn by visual inspection without error bars, so this ambiguity is the single most load-bearing premise for the central claim of four RL-driven phases and the exponential reversal-time scaling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a one-dimensional active Ising model in which each spin updates its orientation via Q-learning: the state is whether the spin is aligned with the local majority, the action is to flip or keep its orientation, and the cost penalizes losing neighbours. Positions are updated by random self-propulsion steps drawn from a uniform distribution. The authors report four phases in the (\\bar{v}, \\varepsilon) plane: disorder, flocking, flipping, and oscillatory. They further characterize the flipping phase by an exponential decay of the mean reversal time with \\varepsilon and a power-law decay with v0, including a data-collapse with fitted exponents. The oscillatory phase is claimed to be chaotic on the basis of a continuous FFT spectrum and a positive largest Lyapunov exponent computed from the order-parameter time series. The abstract states that the RL approach reproduces known 1D flocking phenomena without explicit spin-spin interactions.","tokens_in":14474,"tokens_out":5367,"duration_ms":47797,"significance":"If the results are correct, the paper would show that a genuinely local learning rule, based only on neighbour retention, can generate collective flocking and direction reversals in 1D, and would report a new oscillatory (possibly chaotic) phase. The study is relevant to the growing use of reinforcement learning in active matter. The paper is clearly written and includes a useful visual description of the reversal mechanism (Fig. 6), and the scaling analyses of the reversal time are concrete and falsifiable. However, the central claims rest on the correctness of the nonstandard Q-learning update in Eq. (3) and on the interpretation of the Lyapunov exponent in a stochastic system; both need to be addressed before the conclusions can be accepted.","major_comments":[{"comment":"The Q-learning update as written is not the standard Bellman update. For a cost-minimizing agent with zero discount, the usual update is Q \\leftarrow (1-\\alpha)Q + \\alpha C, whereas Eq. (3) has Q \\leftarrow -Q(1-\\alpha) + \\alpha C. The negative sign causes Q values to alternate in sign on successive visits, and because exploitation in Eq. (4) selects the action with the minimum Q, the policy can artificially switch actions at a frequency set by the visit rate. This could produce the flipping and oscillatory phases without any physical mechanism. If the minus sign is a typo, the published equation does not describe the simulations; if it is not, the learning rule is unconventional and must be justified. In either case, the reproducibility of the entire phase diagram and of the reversal-time scaling is compromised. The authors must clarify or correct Eq. (3) and confirm that all reported results are unaffected.","section":"II, Eq. (3)"},{"comment":"The claim that the oscillatory phase is chaotic is not convincingly supported. The largest Lyapunov exponent is extracted from a single scalar observable (the order parameter) of a many-particle stochastic dynamics via time-delay embedding, but no embedding dimension, delay, or evolution time is reported, and no surrogate-data or other null-hypothesis test is performed. A continuous FFT spectrum and its finite-size dependence are also consistent with stochastic noise, so they do not by themselves establish deterministic chaos. Furthermore, the values in Table I (0.0385\\pm0.0049, 0.0376\\pm0.0082, 0.0368\\pm0.0095) are statistically indistinguishable across the three speeds shown. Please provide a more complete nonlinear time-series analysis, including a discussion of the embedding parameters and a test against surrogate data.","section":"III C, Table I"},{"comment":"The phase diagram is constructed by visual inspection of order-parameter time series and neighbour distributions, with no quantitative criterion defining the phase boundaries and no error estimates. The boundaries are therefore not reproducible from the text. The authors should define explicit classification rules (for example, thresholds on the time-averaged absolute magnetization, cluster-size statistics, or switching frequency) and demonstrate that the diagram is stable under changes in these criteria, in system size, and in the number of realizations.","section":"III A, Fig. 3"},{"comment":"The mean reversal time \\langle T\\rangle is not defined operationally. The paper does not specify how a reversal event is detected from the order-parameter time series, such as a sign change or a threshold crossing, nor whether a dwell-time criterion is used. Different detection algorithms can lead to different fitted exponents in the scaling laws of Figs. 7 and 8. Please give the exact procedure used to extract \\langle T\\rangle.","section":"III B"}],"minor_comments":[{"comment":"The definition of the state S_i(t) does not state how ties are broken when the numbers of neighbouring + and - spins within \\delta x are exactly equal, or what happens if there are no neighbours in the range. Please clarify the convention.","section":"II"},{"comment":"The notation for the self-propulsion speed is inconsistent: the text defines v1 and v2, then v0 as the mean, while Fig. 3 uses \\bar{v}=v1\\Delta t/\\delta x and Fig. 7 uses v0. The paper should explicitly relate these quantities and state the precise values or ranges used for v2/v1 in all simulations, since this ratio affects the distribution of step sizes.","section":"II"},{"comment":"The data collapse in the inset of Fig. 7 is presented without a quantitative measure of collapse quality or error bars on the collapsed points. The exponents \\delta and \\lambda are obtained from separate fits of \\beta and \\gamma; please state the fits' ranges and uncertainties and show the collapse with residuals or a collapse metric.","section":"III B, Fig. 7"},{"comment":"The text says the FFT spectrum shows a 'power law decay' at intermediate frequencies and Fig. 10 labels a slope of approximately 1.0, but the fitted exponent and the frequency range of the fit are not given. Please provide the fit details.","section":"III C, Fig. 10"},{"comment":"The data availability statement mentions data 'upon reasonable request' but does not mention code. Given the ambiguity surrounding Eq. (3) and the simulation-based nature of the results, releasing the source code would substantially improve reproducibility and is strongly recommended.","section":"V"},{"comment":"There are typographical and formatting issues, for example 'Spatiotemporal intermittency (SPI)' in Fig. 9 versus 'STI' in the text, and inconsistent spacing in several equations. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the nonstandard Q-learning update in Eq. (3). I suspect the negative sign may be a typo, but as a referee I cannot assume that; if it is a typo and the simulations used the standard update, the paper needs to state this explicitly and, ideally, provide code to confirm. The chaotic-phase claim is the other weak point; the Lyapunov analysis is too thin. A successful revision would resolve these two points and add quantitative criteria for the phase diagram. I do not see a fundamental flaw in the overall approach, so the paper remains within the scope of corrective major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the idea is workable—1D active Ising spins driven by a Q-learner with a cohesion cost—and the phase diagram is plausible. But one load-bearing technical ambiguity needs to be fixed before I'd trust the new phases, and the chaos claim is under-supported.\n\nWhat's new: combining Q-learning with a 1D active Ising model, with no explicit spin-spin interaction, and finding an oscillatory phase at high speed and low epsilon. That is a genuine extension of Durve et al.'s 2D RL flocking and of the 1D active Ising literature. The flipping-phase mechanism—a small counter-propagating cluster nucleating at the fore-end—is clearly described, and the density/magnetization profiles in Figs. 5-6 are convincing. The exponential decay of mean reversal time with epsilon and the scaling collapse are reasonable numerical results, though fits to one's own simulations always need care.\n\nThe soft spots, in order:\n\n1. Eq. (3) is not a Bellman update. The standard Q-learning rule for a minimization problem is Q <- (1-alpha)Q + alpha*C. The printed minus sign means Q values alternate in sign on successive visits to the same state-action pair. Since Eq. (4) picks argmin Q, this can manufacture artificial switching, which could masquerade as the flipping or oscillatory phases. If the minus sign is a typo, the paper still needs to say so and provide code or at least the corrected update. Without code or data, this is the single most important reproducibility issue.\n\n2. The Lyapunov exponents in Table I are computed from a stochastic time series, but the paper reports no embedding delay, embedding dimension, evolution time, or number of replacement steps. Positive exponents around 0.037 with uncertainties of 0.005-0.01 are fragile without surrogate tests or a check against a known periodic signal. A period-2 oscillation with intermittent bursts can produce spurious positive exponents if the embedding is poor. This is the least supported claim.\n\n3. Phase boundaries in Fig. 3 are drawn visually with no error bars. Minor, but worth stating.\n\nWhat the paper does well: the model is simple, the numerical exploration is systematic (N=1000 and 3000, 100-200 realizations), and the flipping mechanism is physically intuitive. The authors cite the relevant 1D flocking literature properly; self-citations are not load-bearing.\n\nWho it's for: active matter researchers interested in RL-based models and 1D flocking. It deserves a serious referee—the idea is worth engaging—but the referee should insist on a corrected update, code/data, and a properly documented Lyapunov computation.\n\nMy recommendation: send it to peer review with a clear request for major revision. The central ambiguity in Eq. (3) must be resolved before the phases can be trusted.","headline":"A plausible 1D RL flocking model with a promising phase diagram, but the printed Q-update is not standard and the new oscillatory/chaos claims depend on details the paper does not report.","tokens_in":15000,"tokens_out":2713,"would_cite":false,"duration_ms":23574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Q-learning produces four phases in one-dimensional active Ising spins.","keywords":["active Ising model","Q-learning","flocking","one-dimensional active matter","directional switching","chaotic phase","Lyapunov exponent","reinforcement learning"],"falsifier":"Run the same model with the standard Q-learning update Qi ← Qi + α(Ci − Qi) at the same parameters and check whether the four phases, the exponential reversal-time law, and the positive Lyapunov exponent persist; a second decisive check is to record the converged Q-table and test whether its entries alternate sign under the printed update.","tokens_in":14014,"feed_emoji":"🐦","tokens_out":3761,"duration_ms":30839,"temperature":0.7,"pith_summary":"The paper argues that a one-dimensional flock of Ising spins can organize into collective motion without any explicit interaction between spins, purely through each spin's learned response to losing neighbors. Using Q-learning with a cost that penalizes a drop in local neighbor count, the authors map a phase diagram in the plane of mean self-propulsion speed and exploration probability ε. They identify four phases: a disordered one, a flocking phase, a flipping phase in which a condensed flock stochastically reverses direction, and a newly reported oscillatory phase in which magnetization flips at every time step. The reversal time in the flipping phase decays exponentially with ε and obeys a scaling collapse. The oscillatory phase is characterized as chaotic through a continuous power spectrum and a positive Lyapunov exponent.","feed_headline":"Q-learning produces four phases in one-dimensional active Ising spins","feed_subtitle":"A neighbor-count cost alone yields disorder, flocking, flipping, and a chaotic oscillatory phase.","key_machinery":"The central machinery is a per-spin Q-learning scheme in which each Ising spin maintains a 2×2 Q-table mapping its state (aligned or anti-aligned with the majority of neighbors within a range δx) to two actions (flip or keep orientation). The action is chosen by an ε-greedy rule: with probability ε a random action is taken, otherwise the action with minimum Q-value is chosen; ε therefore acts like thermal noise. The Q-table is updated using the cost function Eq. (2), which returns 1 if the spin loses neighbors after moving and 0 otherwise, with learning rate α=0.001. Position updates are taken from a uniform speed distribution, and the absence of any explicit inter-spin interaction is the feature that makes the learned dynamics the sole ordering mechanism.","core_discovery":"The central claim is that reinforcement learning alone—specifically Q-learning with a two-state, two-action table and a cost that is 1 when a spin's neighbor count drops and 0 otherwise—reproduces the known phenomenology of the one-dimensional active Ising model and adds a new phase. No spin-spin coupling term is put in by hand: the only feedback is cohesion, measured by neighbor retention. In the (speed, ε) plane the system exhibits disorder, partial flocking, a flipping phase where the whole condensed flock reverses its direction, and an oscillatory phase where the order parameter changes sign every time step. The flipping phase's mean reversal time decays exponentially with ε and the data collapse under power-law rescalings of both axes. The oscillatory phase shows a continuous Fourier spectrum with a dominant peak at frequency 0.5, and the largest Lyapunov exponent computed from the order-parameter time series is positive, indicating chaos.","pith_inferences":["The paper's logic suggests that cohesion cost alone is a sufficient learning signal for polar order, which would connect RL-driven flocking to contact- and entropy-based explanations of collective motion.","A direct testable extension would be to record the learned Q-values in the oscillatory phase: the period-2 peak suggests the policy settles into a deterministic alternation rule, a limit cycle in policy space rather than in physical coordinates.","Applying the same neighbor-retention cost to agents with continuous orientations should produce a similar disordered-to-flipping crossover as the exploration rate is lowered.","The ratio of hopping step to interaction range appears to control the boundary between flipping and oscillatory behavior, so varying δx at fixed v0 should shift the phase boundaries in a predictable way."],"forward_implications":["The known disorder–flocking–flipping sequence of the 1D active Ising model is obtained without an interaction Hamiltonian, suggesting that cohesion feedback can replace explicit alignment forces.","The exponential decay of the flipping-phase reversal time with ε, together with the power-law collapse, implies a noise-tunable switching clock for the condensed flock.","The newly reported oscillatory phase, with period-2 magnetization flips and a positive Lyapunov exponent, is a candidate for a genuinely learned chaotic collective state.","The same RL framework can be extended to higher dimensions and to learnable hopping step sizes, as the authors note."],"supporting_citations":[{"why":"Supplies the Q-learning and ε-greedy foundations on which the model's update and action rules are built.","marker":"[5]"},{"why":"Provides the Q-learning convergence background that the paper's update equation is meant to instantiate.","marker":"[6]"},{"why":"Defines the one-dimensional lattice flocking model with stochastic directional reversal that this work extends.","marker":"[21]"},{"why":"Supplies prior evidence of directional switching in one-dimensional flocks, the baseline the flipping phase is compared against.","marker":"[23]"},{"why":"Demonstrates that cohesive flocking can be obtained from minimization of neighbor loss using RL, the direct precursor of the cost function used here.","marker":"[34]"},{"why":"Supports interpreting the continuous Fourier spectrum and its dependence on data length as signatures of chaotic dynamics.","marker":"[59]"},{"why":"Provides the Wolf algorithm used to compute the largest Lyapunov exponent.","marker":"[62]"},{"why":"Provides the numerical time-series method used to extract the Lyapunov exponent from the order-parameter data.","marker":"[63]"}],"fun_headline_variants":["Q-learning creates four phases in 1D active Ising model","Reinforcement learning yields chaos and flocking in spins","Neighbor count alone drives four spin phases via Q-learning","RL reveals flipping and chaotic phases in active Ising","One cost function, four phases: Q-learning in spin flocks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulation's Q-learning update, Eq. (3), has a negative sign on the previous Q-value instead of the standard Bellman temporal-difference term, so the reported phases would be artifacts if the code follows the printed equation and undocumented if the code corrects it.","fun_headline_variants_meta":{"raw":{"variants":["Q-learning creates four phases in 1D active Ising model","Reinforcement learning yields chaos and flocking in spins","Neighbor count alone drives four spin phases via Q-learning","RL reveals flipping and chaotic phases in active Ising","One cost function, four phases: Q-learning in spin flocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1538,"prompt_tokens":950,"completion_tokens":588,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":505}},"tokens_in":566,"tokens_out":588,"duration_ms":5340,"temperature":1.0,"reasoning_tokens":505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T06:01:01.247434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same model with the standard Q-learning update Qi ← Qi + α(Ci − Qi) at the same parameters and check whether the four phases, the exponential reversal-time law, and the positive Lyapunov exponent persist; a second decisive check is to record the converged Q-table and test whether its entries alternate sign under the printed update.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Q-learning and ε-greedy foundations on which the model's update and action rules are built."},{"cited_title":"Chat´ e, F","cited_arxiv_id":null,"evidence_quote":"Defines the one-dimensional lattice flocking model with stochastic directional reversal that this work extends."},{"cited_title":"Czir´ ok, A.-L","cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence of directional switching in one-dimensional flocks, the baseline the flipping phase is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that cohesive flocking can be obtained from minimization of neighbor loss using RL, the direct precursor of the cost function used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports interpreting the continuous Fourier spectrum and its dependence on data length as signatures of chaotic dynamics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the numerical time-series method used to extract the Lyapunov exponent from the order-parameter data."}],"review_version":1}