{"id":"0bca3afc-b256-4d88-afc4-db5a15aa3f49","arxiv_id":"1908.02138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A robot learns to follow store norms using norm-based rewards, and the learned abstract policy transfers to a different factory domain without retraining.","lead":"This paper shows how formal social rules, called norms, can be bundled into an institution and turned into automatic rewards for a reinforcement learning robot, so the robot learns to behave politely. The same abstracted rule set is then reused in a new factory environment, transferring the learned behavior without starting from scratch.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central transfer claim lacks direct zero-shot evidence: Fig. 9 compares fine-tuned B2 with from-scratch B3, while B1 is only described verbally in Sec. V-E.","rationale":"The reader identified equivalence-class invariance under re-grounding as the weakest assumption. I agree that this is the conceptual hinge, but the more immediate defect is that the paper never directly measures zero-shot reuse (B1); it quantifies only B2 vs B3. That missing measurement is precisely what would establish or refute the invariance assumption, since B1 is the empirical test of whether the abstract state features remain semantically faithful in the factory. The reported B2 decline and the ambiguity about B3's state representation reinforce the need for this check. The concern is not fatal to the framework -- the proof-of-concept experiments 1, 2, 4, and 5 support the reward-shaping and abstraction components -- but it does place the transfer claim, the paper's headline contribution, on a single qualitative sentence. Hence the reader's CONDITIONAL verdict is appropriate and no adjustment is needed.","tokens_in":20916,"tokens_out":9720,"duration_ms":103365,"concrete_test":"Re-run Sec. V-E scenario (B) and report B1 quantitatively: mean and per-trial success rate (episodes where Forky lifts the grounded box and reaches the hatch within the time limit) and the reward curve over 10 trials, with B1 evaluated before any additional training. Compare B1 against B3 using the same 39-element abstract state representation, and against a control abstract policy trained in Store with norm/action labels scrambled. If B1's success rate is not significantly above the scrambled-policy/random baseline, the transfer claim reduces to initialization-only and Sec. IV-B's invariance assumption is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that the same institution, re-grounded, yields an abstract policy that transfers to a new domain. The decisive experiment is Sec. V-E scenario (B), yet the only quantitative curves in Fig. 9 are B2 (transferred Store policy plus 200k additional training steps in the factory) versus B3 (from scratch). B1 -- direct application of the Store-trained abstract policy to Forky with no further gradient updates -- is reported only as 'in most of the episodes manages to reach the hatch.' No success rate, reward curve, or variance is given. This is load-bearing because B2 confounds two things: the representational transfer claimed in Sec. IV-B (equivalence classes [active], [position], etc. remaining semantically faithful under re-grounding) and 200k steps of target-domain fine-tuning. Moreover, B2's cumulative reward is reported to decline, which is consistent with the transferred policy being a fragile initialization rather than a reusable abstract policy. Without B1, the central claim 'apply the same set of norms over different domains' is not empirically demonstrated; the evidence only shows that an abstract Store policy plus fine-tuning beats learning from scratch (and even that comparison depends on B3's state representation being specified as the same abstract representation, which the paper does not state).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to use a formal model of social institutions to guide reinforcement-learning agents toward normative behavior. An institution is defined as a tuple of roles, actions, artifacts, and norms, and each norm has fulfillment and violation semantics over trajectories. A grounding maps these abstract elements to a concrete domain, and state variables are grouped into equivalence classes such as [active] or [position]. During execution, norm states (fulfilled, violated, neutral) are computed and transformed by a reward function F_norm into RL feedback. The authors introduce \"abstract policies\" trained on the institution-level state space and claim that these policies transfer across domains via re-grounding. Experiments in a simulated store show a robot learning to pick and pay for goods, compare standard and abstract learning, report transfer to a factory domain, evaluate reward-shaping variants, and demonstrate a two-agent seller/buyer scenario.","tokens_in":21182,"tokens_out":5757,"duration_ms":59244,"significance":"If the transfer claims hold, the paper makes a useful bridge between normative multi-agent systems and deep RL: it provides a declarative and reusable way to encode social knowledge, converts norm semantics into automatic reward shaping, and reduces the state space through institution-level abstraction. The formal model is clearly presented, the framework is RL-algorithm agnostic, and the multi-agent extension is a genuine plus. The empirical study is best read as a proof of concept; the main weakness is that the decisive zero-shot transfer result (B1) is not quantitatively reported, so the central \"apply the same set of norms over different domains\" claim is not yet demonstrated at the level of the paper's own headline. The norm-state reward is constructed from the same semantics that define fulfillment and violation, which is a legitimate reward-shaping design rather than a hidden circularity.","major_comments":[{"comment":"The central transfer claim is not quantitatively supported because B1, the zero-shot application of the Store-trained abstract policy to Forky, is reported only as a qualitative statement (\"in most of the episodes manages to reach the hatch\"), with no success rate, reward curve, or variance. Figure 9 compares B2 (transferred policy plus 200k additional training steps) with B3 (from scratch), and B2's cumulative reward declines; this comparison cannot separate the representational transfer claimed in Section IV-B from target-domain fine-tuning. Please report B1's quantitative behavior (e.g., episode success rate and return as a function of episodes) or revise the conclusion to claim only fine-tuned transfer.","section":"Section V-E, Fig. 9"},{"comment":"It is not stated whether B3 (from scratch) uses the same abstract institution-level state representation as B2 or the full 162-element state space described in Section V-B. If B3 uses the full representation, the comparison conflates the effect of abstraction with the effect of transfer, and the claim that \"learning from scratch did not manage to achieve any significant results\" is not a controlled baseline. Specify B3's observation and action spaces, and if necessary rerun B3 with the abstract representation.","section":"Section V-E, Fig. 9"},{"comment":"The claim that abstract learning is \"significantly faster\" rests on mean curves without error bars or confidence intervals, and Fig. 7b compares only the best trial from each setting. Since RL runs have high variance, report the full distribution (e.g., interquartile ranges, per-trial success rates at a fixed step budget) and define the \"success\" criterion used to classify trials as converged or failed.","section":"Section V-D, Fig. 7"}],"minor_comments":[{"comment":"The hyperparameters were chosen via pilot tests that are not reported; for reproducibility, include the full hyperparameter set (including entropy coefficient, learning rate schedule if any, and number of seeds) in an appendix, and ideally a sensitivity check on the reward-shaping constants.","section":"Section V-B, Table I"},{"comment":"The sentence \"Mean line represents the learning curve averaged over 10 independent training trials, where for each training trial data is collected over 16 parallel simulations\" should specify how the 16 parallel simulations are aggregated before averaging over the 10 trials.","section":"Section V-C"},{"comment":"The phrase \"in the trail (A)\" should read \"in the trial (A)\"; the same typo appears elsewhere in the experiment descriptions.","section":"Section V-E"},{"comment":"There is a duplicated article in \"the the credit assignment problem\" in the first sentence of Section II.","section":"Section II"},{"comment":"The row for \"Abstract Learning - Success Failed (Learning) Success (Transfer)\" is syntactically confusing and should be reformatted, for example into separate rows for \"Abstract, no shaping\" and \"Abstract with transfer\".","section":"Table II"},{"comment":"The norm-state reward values are specified as 1.0/(number of norms) for the (n,f) transition, but the paper does not state how multiple norms are combined when several transitions occur in the same step, nor whether a norm that goes directly from neutral to violated receives the same penalty as one that was previously fulfilled; please make this explicit for reproducibility.","section":"Section V-F"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from being positioned explicitly as a proof of concept rather than a definitive empirical demonstration; the missing B1 data is fixable but central. The formal framework relies heavily on the authors' prior work [24], and the paper should be self-contained enough for a general RL audience. The cognitive-science and linguistics discussion is speculative and could be shortened without harming the core contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The institutional-reward idea is the real contribution: encoding norms as institution-level semantics and using norm-state transitions as reward signals is a clean way to shape RL, and the equivalence-class abstraction (state variables like [active], [position]) is a sensible mechanism for learning policies that might transfer. The formal definitions are careful, and the experimental setup is honest about what the authors did. That said, the headline claim—transfer of the same set of norms across domains—is not actually backed by quantitative evidence. The stress-test note is right: in Experiment 3, the only curves in Fig. 9 are B2 (transferred Store policy plus 200k fine-tuning steps) versus B3 (from scratch). B1, the direct zero-shot application of the Store policy to Forky, is reported only as “in most episodes manages to reach the hatch.” No success rate, reward curve, or variance. That is load-bearing, because B2 confounds representational transfer with target-domain fine-tuning, and the fact that B2's cumulative reward declines is consistent with the transferred policy being a fragile initialization rather than a genuinely reusable abstract policy.\n\nOther soft spots are minor but real: no error bars on the learning curves, Experiment 2 compares best trials, hyperparameters are from unreported pilot testing, no code is released, and the “not dependent on a particular RL algorithm” claim is overstated given only PPO is tested (the conclusion does hedge, but the abstract doesn't). None of these break the framework's conceptual value; they weaken the empirical support.\n\nWhat the paper does well: the norm-state reward shaping (Experiment 4) is nicely demonstrated, the multi-agent extension is a useful bonus, and the abstraction idea is clearly explained. The self-citation to the prior institution paper [24] is legitimate, not padding.\n\nWho is this for? Researchers working on normative RL, reward shaping, and abstraction in robotics. They will get a clear formalism and a promising direction, but they should not cite the transfer result as demonstrated. I'd like to see a B1 curve before taking that claim seriously.\n\nVerdict: deserves a serious referee, but the referee should ask for the B1 data (or at least a success-rate metric over many episodes) before the transfer claim stands. If the authors can't produce it, the paper should be reframed as a proof-of-concept for norm-based reward shaping with a speculative transfer mechanism.","headline":"Norm-guided reward shaping is a solid, clearly formalized idea, but the paper's headline claim of cross-domain transfer rests on a single qualitative result with no zero-shot curve.","tokens_in":21652,"tokens_out":3747,"would_cite":true,"duration_ms":36438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Social norms encoded as an institution can automatically shape robot rewards, and the resulting abstract policy transfers to a different robot and environment via re-grounding.","keywords":["social norms","institutions","reinforcement learning","reward shaping","transfer learning","abstract policies","grounding","normative behavior"],"falsifier":"Take a norm that requires distinguishing two objects that share the same state-variable classes (same active and position features) but have different normative status, say one object is 'goods' to pay for and another visually identical object must not be taken. If the original institution never had a feature class for this distinction, the abstract policy re-grounded onto the new domain will treat both objects the same and violate the norm; observing this failure would refute the claim that re-grounding preserves the normative semantics.","tokens_in":20745,"feed_emoji":"🤖","tokens_out":4224,"duration_ms":40807,"temperature":0.7,"pith_summary":"This paper claims that social norms, written down once in an abstract institution (roles, actions, artifacts, norms), can be used to automatically shape the rewards of a reinforcement-learning agent so it learns to behave in a socially acceptable way. The same institution can then be re-grounded onto a different robot, body, and environment, yielding an abstract policy that transfers without retraining. The authors demonstrate this with a store scenario in which robot Robby learns to pick a battery and pay before leaving, and a factory scenario in which robot Forky applies the same learned pattern to lift a box and leave it at a hatch. A policy learned from scratch in the factory mostly failed within the training budget, while the transferred policy succeeded, which is the empirical core of the claim.","feed_headline":"Same social norms teach two different robots their jobs","feed_subtitle":"Robby's store-learned policy, re-grounded as an abstraction, lets factory robot Forky sort boxes without retraining.","key_machinery":"The central object is the institution: a tuple $\\langle Arts, Roles, Acts, Norms\\rangle$, with each norm a qualifier applied to role–action–artifact triples, e.g., $\\text{must}(\\text{Buyer}, \\text{Pay}, \\text{Cash})$. The mechanism is grounding—a mapping from these abstract categories to concrete agents, behaviors, and objects—together with norm semantics that classify any execution trajectory as fulfilling, violating, or neutral with respect to each norm. This classification drives the reward-shaping function $F_{norm}$ over norm-state transitions, and the same semantics defines equivalence classes of state variables that yield an abstract input vector for the policy, reducing the store's full 162-element observation to 39 elements.","core_discovery":"The central claim is that an institution's declarative norms—statements like 'a buyer must use the goods and pay at the pay place before leaving'—can be given execution-level semantics as sets of trajectories, and this semantics provides everything needed to (a) build an automatic reward signal that steers learning toward adherence, and (b) define an abstract state and action space based on equivalence classes of state variables ([active], [position], etc.) that stays stable when the institution is grounded onto new agents, behaviors, and objects. Learning at this institutional level produces an abstract policy that encodes procedural knowledge of how to fulfill the norms, and that policy transfers to a different domain with the same categories. In the paper's experiments, standard learning in the large store state space failed in almost all trials within four million steps, while abstract learning converged; and the store-learned abstract policy, directly re-grounded, let the factory robot perform the normative task in a dynamic environment where learning from scratch did not achieve significant results.","pith_inferences":["If primitive norm policies (one per qualifier like 'at' or 'before') can be learned and recombined, they could serve as reusable, composable skills for an agent, effectively building procedural knowledge from declarative pieces.","The equivalence-class abstraction suggests a testable connection to schema theory: the abstract policy may predict that agents re-ground faster when the new domain shares the same sensorimotor features, a hypothesis that could be measured in human or robotic learning experiments.","The approach could be extended to automatically search for admissible groundings during learning, which would make the method data-efficient in novel environments by rewarding any trajectory that is adherent under some grounding.","Hierarchically grounding norms in terms of other norms, as the authors sketch for 'before' defined over two 'must' norms, points toward compositional norm understanding, but this is only implicitly explored in the paper."],"forward_implications":["Normative reward shaping can be built automatically from a declarative norm specification, removing the need to hand-engineer dense reward functions.","Abstract policies learned at the institution level reduce the observation space and make learning feasible where full-state learning fails.","A policy trained in a simple simulator can be transferred to a dynamically complex domain by re-grounding, which is a route to sim-to-real for normative behavior.","The same mechanism extends to multi-agent settings: agents grounded to different roles receive role-specific norm feedback and learn to cooperate on shared norms.","The method is not tied to a specific RL algorithm, since the institution supplies only a feedback signal."],"supporting_citations":[{"why":"Supplies the full formal institution model, grounding, and verification mechanisms that the paper builds on.","marker":"[24]"},{"why":"Provides the reward-shaping background that motivates the norm-state feedback signal, though the paper does not prove policy invariance.","marker":"[14]"},{"why":"The PPO algorithm used to train the simulated agents in all experiments.","marker":"[32]"},{"why":"The Unity game engine used to simulate the store and factory domains.","marker":"[31]"},{"why":"The ML toolkit and bridge used to run PPO on simulation data.","marker":"[33]"}],"fun_headline_variants":["Norm-based policy transfers from store robot to factory bot","Social norms give robots a transferable learning shortcut","One abstract norm policy, two very different robot jobs","Institutional norms as a portable guide for robot learning","Learn norms once, reuse the policy on any robot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transfer works only if the equivalence classes that the abstract policy sees—things like 'active' and 'position'—carry the same meaning in the new domain as they did in the original one, for the new agents, behaviors, and objects.","fun_headline_variants_meta":{"raw":{"variants":["Norm-based policy transfers from store robot to factory bot","Social norms give robots a transferable learning shortcut","One abstract norm policy, two very different robot jobs","Institutional norms as a portable guide for robot learning","Learn norms once, reuse the policy on any robot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1609,"prompt_tokens":907,"completion_tokens":702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":523,"tokens_out":702,"duration_ms":8180,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:39:39.746560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a norm that requires distinguishing two objects that share the same state-variable classes (same active and position features) but have different normative status, say one object is 'goods' to pay for and another visually identical object must not be taken. If the original institution never had a feature class for this distinction, the abstract policy re-grounded onto the new domain will treat both objects the same and violate the norm; observing this failure would refute the claim that re-grounding preserves the normative semantics.","supporting_citations":[{"cited_title":"Norms, Institutions, and Robots","cited_arxiv_id":"1807.11456","evidence_quote":"Supplies the full formal institution model, grounding, and verification mechanisms that the paper builds on."},{"cited_title":"Policy invariance under reward transformations: Theory and application to reward shaping,","cited_arxiv_id":null,"evidence_quote":"Provides the reward-shaping background that motivates the norm-state feedback signal, though the paper does not prove policy invariance."},{"cited_title":"https://unity.com/,","cited_arxiv_id":null,"evidence_quote":"The Unity game engine used to simulate the store and factory domains."}],"review_version":1}