{"id":"3fd8ac36-3d7e-4056-80c7-01a66ca27081","arxiv_id":"2506.01016","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Optimistic critic estimates (mean or max over two Q functions) counteract value underestimation and recover most of the performance lost by shrinking the actor in SAC and DrQ.","lead":"Smaller actor networks in Soft Actor-Critic and DrQ reinforcement learning agents lose performance and overfit their critics, and the paper traces this to poor data collection caused by value underestimation. Replacing the minimum of two critic estimates with their mean or maximum largely rescues the small actors, a simple fix for deploying compact policies in resource-constrained settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim that poor data collection is the main driver is not isolated from the direct change in the actor's objective.","rationale":"The reader's weakest assumption correctly identifies the causal attribution issue: the paper does not isolate the data-collection channel from the direct change in the actor's learning objective when switching from min to mean/max critics. The evidence in Sections 3.1 and 3.2 is correlational (data-quality correlations, lower Q-values and entropy), and the intervention in Section 4.1 alters both the critic target and the actor's policy-gradient objective. The offline data-swap experiment (Fig. 2) shows only partial recovery in some SAC environments and no clear effect in DrQ (Fig. 9), which does not establish that poor data collection is the main cause. The paper itself flags lack of hyperparameter search in Limitations, and the only tuned parameter (L2-from-init) is tuned on a single environment, which weakens the auxiliary claim that critic regularization is ineffective and hence that optimism is special. These issues do not undermine the core empirical finding that mean/max critics can improve small-actor performance, but they make the mechanistic 'mostly due to' claim stronger than the evidence. A controlled offline data-quality experiment would settle whether the data-collection channel is sufficient; absent that, the verdict should remain CONDITIONAL rather than full ACCEPT, consistent with the reader's recommendation.","tokens_in":15538,"tokens_out":1652,"duration_ms":15167,"concrete_test":"Run a controlled experiment on cheetah-run and hopper-hop isolating the data-collection channel: train the xs actor offline on a fixed, high-quality replay buffer collected by a regular r actor, with the critic and actor update rules unchanged (min rule). If the xs actor trained on r-collected data reaches performance comparable to the mean/max intervention (e.g., within 20% of the r-actor's final return), the data-collection channel is sufficient and the causal claim holds. If it does not recover, then the mean/max rescue works primarily through directly altering the actor's objective rather than through data quality, and the 'mostly due to' claim should be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that small-actor performance loss is 'mostly' caused by poor data collection due to value underestimation, and that mean/max critics rescue performance by alleviating this underestimation. The evidence in Section 4.1 replaces min with mean/max in both the critic target (Eq. 4, V_bar) and the actor objective (Eq. 5). This changes the learning signal for the actor directly, not only the data it collects. The paper's own Section 3.1 experiments (Fig. 2) show that a regular actor trained on small-actor data loses performance, and the xs actor trained on regular-actor data improves only in some SAC environments and not in DrQ (Fig. 9). These data-quality results are correlational and incomplete for the causal channel: they do not demonstrate that a small actor trained on high-quality data from a regular actor fully recovers performance, nor do they measure state coverage. Section 3.2 adds another correlation (smaller actors have lower Q and entropy). Because the mean/max intervention also changes the actor's policy-gradient objective and entropy bonus coupling, the 'mainly due to data collection' attribution is underdetermined: the improvement could come substantially from directly counteracting the min-clipping bias in the actor's gradient, reducing pessimism in action selection without any change in data distribution. The paper's Limitations section acknowledges default hyperparameters are used and no hyperparameter search is performed, and the L2-from-init coefficient is tuned on one environment only (Section 4.2), which weakens the claim that critic regularization fails to rescue performance and thereby supports the specificity of the optimism intervention.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically studies what happens when the actor network in Soft Actor-Critic (SAC, state-based) and DrQ (pixel-based) is made much smaller than the critic on DeepMind Control tasks. It documents that smaller actors degrade final performance, are associated with higher critic overfitting (o_phi, dormant neurons), lower relative Q-values, and lower policy entropy. Data-quality experiments show that a regular-sized actor trained on data collected by a smaller actor loses performance, and that in some SAC environments a small actor improves when trained on data from a regular actor, while DrQ shows no clear transfer benefit. The central intervention replaces the minimum over the two critics with their mean or maximum in both the critic target and the actor objective; this substantially improves small-actor performance in SAC and, to a lesser extent, in DrQ, whereas critic regularization and resets do not. The paper concludes that poor data collection due to value underestimation is a main cause of small-actor degradation and that alleviating underestimation can empower small actors.","tokens_in":15736,"tokens_out":5275,"duration_ms":54381,"significance":"The main positive result---that mean/max critics can markedly improve the performance of small actors---is practically relevant for deployment-oriented actor-critic methods, and the paper presents it with a broad set of environments, 10--20 seeds, and confidence intervals. The authors also report null or weak results for DrQ and for critic-regularization baselines, which is good empirical practice. The paper ships no code, but the experimental setup is described in enough detail to be reproducible from existing SAC/DrQ implementations. If the causal story were fully established, the work would also contribute to the understanding of value bias in actor-critic methods. However, as it stands, the evidence for the 'value underestimation leads to poor data collection' mechanism is not isolated, and one piece of evidence used for underestimation is confounded. These issues do not invalidate the empirical intervention, but they do affect the strength of the explanatory claims.","major_comments":[{"comment":"The claim that smaller actors 'result in Q-value underestimation' is not supported by the presented evidence. The figure plots average critic Q-values relative to the regular actor, but a smaller actor with lower true returns will also have lower unbiased Q-estimates; the relative comparison therefore does not demonstrate underestimation. To support this mechanistic claim, the authors should compare Q-estimates to empirical (Monte Carlo) returns or examine the sign and magnitude of TD residuals. Without such a comparison, the term 'underestimation' is an interpretation rather than a measurement, and it is load-bearing for the paper's causal narrative.","section":"Section 3.2, Fig. 3"},{"comment":"The mean/max intervention replaces the min operator in both the critic target (Eq. 4) and the actor objective (Eq. 5). This changes the actor's policy-gradient signal directly, by removing the min-clipping bias in action selection and by altering the coupling with the entropy bonus, so the observed improvement cannot be attributed solely to improved data collection. The data-quality experiments in Section 3.1 (Fig. 2 and Fig. 9) are correlational and partial: they do not show that a small actor trained on high-quality data from a regular actor fully recovers performance, and they do not measure state coverage. To make the 'mostly due to poor data collection' claim, please add an experiment that varies data collection while keeping the min rule (for example, training the xs actor offline on a buffer collected by the r actor), or rephrase the conclusions to distinguish the empirical intervention from the causal mechanism.","section":"Section 4.1, Eqs. (4)-(5)"},{"comment":"The causal attribution is stated more strongly in the abstract ('poor data collection, due to value underestimation, as one of the main causes') and in the opening of Section 4 ('largely due to poor data collection') than the evidence supports, given the issues in the previous two comments. If the isolating experiment is not added, the authors should soften these statements to say that the results are consistent with this mechanism rather than that the mechanism is established.","section":"Abstract and Section 4 (opening paragraph)"}],"minor_comments":[{"comment":"The caption says 'The top row displays final performance while the bottom two rows display performance throughout training,' but the figure has three rows and the text refers to top/middle/bottom rows with different groupings for SAC and DrQ; please clarify which row corresponds to which setting.","section":"Fig. 4 caption"},{"comment":"The TD error in Eq. (1) is defined with an absolute value, but later the notation T D_phi is used in the definition of o_phi without a formal definition; please define T D_phi explicitly to avoid ambiguity.","section":"Section 2, Eq. (1)"},{"comment":"The labels r/m/s/xs are introduced only in Section 3, but they would be helpful in Section 2.1 where the architectures are described; consider defining them earlier or adding a pointer.","section":"Section 3, actor size labels"},{"comment":"The L2 distance from initialization coefficient is tuned on a single environment (quadruped-run) and then used elsewhere; please state the chosen value and clarify that this tuning is an exception to the otherwise default hyperparameters, as this affects the interpretation of the regularization results.","section":"Section 4.2, L2 distance from initialization"},{"comment":"The figure labels 'online' and 'offline' are not defined in the caption; please clarify whether 'offline' refers to training on a fixed buffer collected by the regular actor and 'online' to the standard interactive training.","section":"Fig. 9 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a practically useful intervention result. My recommendation of major_revision is driven by the two load-bearing issues: the underestimation evidence in Section 3.2 is confounded, and the data-collection attribution in Section 4.1 is not isolated from the direct change in the actor's objective. Both are fixable either by adding a targeted experiment (e.g., offline training of the small actor on regular-actor data with the min rule) or by softening the causal claims. I do not see a fundamental correctness problem with the main empirical finding that mean/max critics help small actors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper maps out when small actors fail in SAC/DrQ and shows that swapping the min of the two critics for mean or max recovers most of the lost performance. That rescue effect is real, consistent across several DMC tasks, and the paper gives the best diagnosis I've seen for why small actors struggle: the critics overfit and value estimates collapse, and the conservatism of the min-operator seems to be the main culprit.\n\nThe systematic measurement is the real contribution. The mean/max combination is not new, but the careful connection between actor shrinkage, o_phi overfitting, dormant neurons, Q underestimation, and the intervention is well done. Ten to twenty seeds with CIs, null results reported (critic regularization, DrQ transfer), and an honest limitations section. That is good empirical craft.\n\nNow the soft spot. The paper claims the performance drop is \"mostly\" caused by poor data collection via value underestimation. That attribution is underdetermined. Replacing min with mean/max changes both the critic target (Eq 4) and the actor's policy gradient (Eq 5), so the improvement could come directly from reducing pessimism in the actor's objective, not only from collecting better data. The transfer experiments are correlational and incomplete: a regular actor trained on small-actor data does lose performance, but the smallest actor trained on regular-actor data only partially recovers in some SAC environments and not at all in DrQ. There is no state coverage measurement. The paper's own evidence supports \"contributes\" more than \"mostly.\" I would rewrite those causal claims and add a cleaner test, e.g., a small actor trained with mean/max critics but forced to use data from the regular actor, or a study of state coverage.\n\nMinor: no code or data release, and the L2 coefficient is tuned on a single environment, so the failure of critic regularization is not a strong negative result. But these are minor.\n\nWho this is for: anyone doing embedded/robotics RL who wants to deploy small actors, and anyone working on alleviating estimation bias in actor-critic methods. The title claim is slightly overstrong, but the underlying recipe is practical and the empirical map is worth having.\n\nRecommendation: send it to peer review, conditional accept. Have the authors tone down the causal attribution, and ideally add the missing control experiment. The core empirical result will survive.","headline":"A careful empirical study on small actors that overclaims the data-collection mechanism; the mean/max rescue is real and the causal story is underdetermined.","tokens_in":16349,"tokens_out":2136,"would_cite":true,"duration_ms":22423,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that shrinking an actor in SAC or DrQ degrades performance mostly because the minimum-of-two-critics rule makes the critic underestimate values, and that replacing that min with the mean or max of the two critics largely…","keywords":["actor-critic","soft actor-critic","small actors","asymmetric actor-critic","value underestimation","optimistic critics","continuous control","reinforcement learning"],"falsifier":"Train the smallest actor under the standard min-rule critic on a fixed dataset collected by a full-size actor, as in the paper's bottom row of figure 2. If this data-sufficient small actor does not recover most of the performance gap, the claim that poor data collection is the main channel would be contradicted.","tokens_in":15279,"feed_emoji":"🤖","tokens_out":3975,"duration_ms":37706,"temperature":0.7,"pith_summary":"The paper argues that the severe performance drop from drastically shrinking the actor in Soft Actor-Critic and DrQ is not a hard limit of the actor's reduced capacity but rather a side effect of value underestimation in the critics. When the actor shrinks, the standard minimum-of-two-critics rule produces overly conservative Q-estimates, the actor under-explores, and the data it collects becomes poor, causing the critics to overfit. The central intervention is simple: replace the min with the mean or the maximum of the two critic Q-estimates in both the critic and actor losses, which raises value estimates, restores policy entropy, and makes small actors competitive with regular-sized actors. If this holds, it gives a cheap, practical recipe for training small, deployable actors in resource-constrained settings without giving up performance.","feed_headline":"Swapping min for mean critics rescues tiny RL actors","feed_subtitle":"Shrinking an actor to 1% of its size stops hurting when critic estimates are combined via mean or max.","key_machinery":"The central object is the aggregation operator over the two critic heads in the SAC/DrQ losses, equations (4) and (5): the standard min, versus the proposed mean or max. This operator sets the target values and the actor-gradient signal; the paper shows that varying only this operator, while keeping network sizes fixed, substantially alters value estimates and downstream data quality.","core_discovery":"On the paper's own terms, the central discovery is that the min-operation used by SAC and DrQ to combine two critic heads is the key bottleneck when the actor is small: it under-estimates Q-values, the actor under-explores as a result, and the data it collects is poor, which in turn makes the critics overfit. Replacing min with the mean or the maximum of the two Q-estimates in both the critic and actor losses raises validation Q-values, restores entropy, and boosts final performance of actors with only 1% of the default parameters; in some environments the small actor with mean/max critics even beats the full actor trained with the min rule.","pith_inferences":["The paper's causal story, that poor data collection via underestimation is the main channel, is supported by correlations and interventions but not fully isolated; a direct test would train a small actor with the min-critic on data from a regular actor and compare recovery, which the paper only partially does.","The finding likely extends to other min-based actor-critic algorithms such as TD3, though those are not tested here; the same swap to mean or max could relieve their underestimation bias in constrained settings.","In sample-scarce or expensive environments, using optimistic critics may be a more effective remedy for small actors than adding regularization or resets, since it directly targets the value bias rather than its consequences.","A testable extension would sweep between min and max (e.g., a weighted average of the two critics) to characterize the bias-variance trade-off and find the optimal optimism level for a given actor size."],"forward_implications":["Small actors (down to 1% of baseline parameters) can be trained with a one-line modification, mean or max critics, to reach near or even above regular-actor performance.","Asymmetric actor-critic with a small, deployable actor becomes more practical, preserving inference-cost and interpretability benefits without a large performance penalty.","The mean/max fix also helps when the actor receives limited inputs (partial observability), suggesting that addressing underestimation is a general strategy for constrained actors.","The findings question the automatic use of the min rule in conservative actor-critic methods, motivating more nuanced bias correction rather than blind overestimation avoidance.","Critic regularization alone does not recover the performance, highlighting that the value-bias channel, not general overfitting, is what matters most for small actors."],"supporting_citations":[{"why":"Supplies the SAC algorithm and the standard min-of-two-critics losses that the paper modifies.","marker":"Haarnoja et al., 2018b"},{"why":"Supplies the DrQ pixel-based baseline and the data-augmentation recipe used in the image experiments.","marker":"Yarats et al., 2021"},{"why":"Provides the motivation for smaller actors and the claim that actor capacity can be reduced while keeping critic size.","marker":"Mysore et al., 2021"},{"why":"Frames the pessimistic exploration problem caused by conservative value estimates, which the paper builds on.","marker":"Ciosek et al., 2019"},{"why":"Supplies the overfitting metric o_phi used to measure critic overfitting with smaller actors.","marker":"Nauman et al., 2024"},{"why":"Supplies the primacy-bias framing and the reset technique tested as a mitigation in the paper.","marker":"Nikishin et al., 2022"},{"why":"Establishes the min-rule tradition in actor-critic methods that the paper's intervention contrasts with.","marker":"Fujimoto et al., 2018"},{"why":"Provides the asymmetric-information bias-correction idea that the paper adapts via actor-latent concatenation.","marker":"Baisero & Amato, 2022"}],"fun_headline_variants":["Mean beats min for tiny actors in RL","Small actors thrive when critics use mean","Fix value underestimation to empower small RL actors","Don't min your critics when actor is small","Rescuing small RL actors by rethinking critic heads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the performance drop from small actors is caused by value underestimation leading to poor data collection, rather than by the small actor's reduced representational capacity, since the evidence is correlational and the main intervention changes the actor's learning objective as well as the data it collects.","fun_headline_variants_meta":{"raw":{"variants":["Mean beats min for tiny actors in RL","Small actors thrive when critics use mean","Fix value underestimation to empower small RL actors","Don't min your critics when actor is small","Rescuing small RL actors by rethinking critic heads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000427,"raw_usage":{"total_tokens":2113,"prompt_tokens":803,"completion_tokens":1310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":1250}},"tokens_in":419,"tokens_out":1310,"duration_ms":11017,"temperature":1.0,"reasoning_tokens":1250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:52:20.595208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the smallest actor under the standard min-rule critic on a fixed dataset collected by a full-size actor, as in the paper's bottom row of figure 2. If this data-sufficient small actor does not recover most of the performance gap, the claim that poor data collection is the main channel would be contradicted.","supporting_citations":[{"cited_title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","cited_arxiv_id":null,"evidence_quote":"Supplies the DrQ pixel-based baseline and the data-augmentation recipe used in the image experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the motivation for smaller actors and the claim that actor capacity can be reduced while keeping critic size."},{"cited_title":"Better exploration with optimistic actor critic","cited_arxiv_id":null,"evidence_quote":"Frames the pessimistic exploration problem caused by conservative value estimates, which the paper builds on."},{"cited_title":"Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the overfitting metric o_phi used to measure critic overfitting with smaller actors."},{"cited_title":"The Primacy Bias in Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the primacy-bias framing and the reset technique tested as a mitigation in the paper."},{"cited_title":"Addressing function approximation error in actor-critic methods","cited_arxiv_id":null,"evidence_quote":"Establishes the min-rule tradition in actor-critic methods that the paper's intervention contrasts with."}],"review_version":1}