{"id":"adc0d399-03ab-45ce-b36e-ca18139fe891","arxiv_id":"2411.11761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A conceptual framework classifies human feedback to RL agents along nine dimensions and seven quality criteria, unifying human-centered, interface-centered, and model-centered design perspectives.","lead":"Researchers propose a nine-dimension framework that maps the many ways humans can give feedback to AI agents, from thumbs-up ratings to corrections, demonstrations, and gaze. The goal is to give the human-computer interaction and reinforcement learning communities a shared language for designing and studying feedback systems beyond today's standard pairwise comparisons.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scalar-bottleneck assumption in §3.2 is the load-bearing step: semantically distinct natural-language and corrective feedback can collapse to the same (T, r_fb) triple, and the paper's own [113] undercuts the projection without being integrated.","rationale":"The reader's weakest assumption is the scalar bottleneck, and I agree that it is the most load-bearing issue. The paper's strongest claim, that the nine dimensions exhaustively and orthogonally classify human feedback and unify human-, interface-, and model-centered aspects, depends on a single formal object F: T -> r_fb. If that object cannot faithfully carry the content of instructive, descriptive, and corrective feedback, then the taxonomy still provides a useful vocabulary but not the claimed unifying abstraction. The concern is grounded in the paper's own examples: 'Clean it using a different type of cleaner' would have to become a target plus a scalar, yet the target is not a reward value and no procedure is given for extracting it. The self-citation [113] is real evidence that the authors are aware of the misspecification risk, but the paper does not address it, so the central claim outruns the support. I do not see this as fatal: the framework is explicitly presented as a conceptual map and the authors honestly note alternative framings in Section 6. The right response is to keep the CONDITIONAL verdict and require the proposed reconstruction test or an equivalent demonstration that the formalization is lossless for non-evaluative feedback. No ad hominem or manufactured issue: this is an internal gap between the formalism and the examples.","tokens_in":48657,"tokens_out":7335,"duration_ms":80857,"concrete_test":"Select all natural-language and physical-correction feedback papers from the survey (e.g., [56, 63, 116, 164, 178, 179]). For each of 20 randomly sampled raw utterances m, have two independent annotators apply Section 3.2's translation literally to produce (T, r_fb, C). Then run a blind reconstruction task: a third annotator must recover the original corrective intent from the triple alone (for a robot plan, the exact corrected action or feature; for text, the exact edit). If reconstruction accuracy is not near-perfect for semantically loaded utterances, the scalar-projection assumption is lossy and the claimed unification of feedback types in Section 3.2 fails as stated. This directly tests the one assumption on which the taxonomy's unifying claim rests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines processed feedback as F: T -> r_fb in R and a translation phi: m -> (T, r_fb, C). The central unifying claim requires every surveyed feedback type to fit this schema without loss. The paper's own formal definitions show the strain. For instructive feedback (D1) the content is carried by the selected target, e.g., a hypothetical trajectory or a feature like 'use a different cleaner', while r_fb is reduced to a constant or confidence value; likewise corrective language feedback from [56, 116, 164] contains propositions ('the mop wet the carpet', 'do this instead') that have no location in a scalar reward. The framework provides no rule for constructing T from arbitrary linguistic or physical-correction measurements, and C is defined only from contextual measurements about the feedback state, not from the intrinsic utterance. Thus two semantically different messages can map to the same (T, r_fb, C) triple, so the formalization does not establish lossless unification; it assumes it. The paper itself cites [113] ('Humans are not Boltzmann Distributions') arguing that modeling humans as noisy scalar rewarders is a misspecification, but does not integrate this objection or bound the information loss. This does not destroy the taxonomy as a design vocabulary, but it is the linchpin of the exhaustiveness/unification claim and is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conceptual framework for human feedback in reward-based reinforcement learning. It introduces a taxonomy of feedback along nine dimensions (human-centered: intent, expression form, engagement; interface-centered: target relation, content level, target actuality; model-centered: temporal granularity, choice set size, exclusivity), along with seven quality metrics (expressiveness, ease, definiteness, context independence, precision, unbiasedness, informativeness). It formalizes feedback as a measurement m translated via a function phi into processed feedback F: T -> r_fb, optionally conditioned on a context encoding C, and derives system requirements for user interfaces, feedback processors, and reward models. The framework is supported by a survey of 141 papers classified in Appendix A and by an implemented prototype interface.","tokens_in":48929,"tokens_out":4498,"duration_ms":42855,"significance":"If taken as a design vocabulary rather than as a lossless formal characterization, the framework is a useful interdisciplinary contribution: it synthesizes a large and heterogeneous literature, connects HCI concerns (cognitive load, expressiveness) with ML concerns (reward-model learnability), and makes concrete, falsifiable design recommendations. The authors deserve credit for the extensive classification appendix, the explicit formalization that builds on the reward-rational choice framework [80], and the derived design requirements (UI.R1-R4, FP.R1-R4, RM.R1-R3), which are concrete enough to guide system builders. However, the paper's stronger claims—that the taxonomy is exhaustive and that diverse feedback types can be unified without loss—are not established by the evidence presented. The iterative methodology in Section 3.1 guarantees classifiability by construction, and the scalar-reward formalism in Section 3.2 assumes, rather than demonstrates, that language, corrections, and demonstrations can be faithfully rendered as (target, scalar value, context) triples. The framework's value as a design space is real; its value as a unifying formal model is overstated.","major_comments":[{"comment":"The central unifying claim depends on the assumption that any feedback measurement can be losslessly represented as a target T, a scalar reward value r_fb, and a context encoding C (Eq. \"F : T → r_fb ∈ R\" and \"φ : m → (T, r_fb, C)\"). This is not established. For instructive and corrective language feedback (e.g., [56, 116, 164]), the propositional content (\"use a different cleaner\", \"do this instead\") has no natural location in a scalar r_fb, and the paper provides no rule for constructing T from arbitrary linguistic or physical-correction measurements. Context encoding C is defined only from contextual measurements m_ctxt (Section 3.2, paragraph \"Based on our definition, we may identify two types of measurement variables\"), not from the intrinsic content of the utterance. Consequently, two semantically distinct messages can map to the same (T, r_fb, C) triple, so the formalization assumes lossless unification rather than demonstrating it. The paper itself cites [113] (\"Humans are not Boltzmann Distributions\") arguing that reducing human feedback to scalar reward is a misspecification, but this objection is not integrated into the formalism or used to bound the information loss.","section":"§3.2"},{"comment":"The taxonomy's exhaustiveness is guaranteed by construction. The authors specify \"Exhaustiveness: All surveyed papers describing types of human feedback for agent training must be classifiable with the given dimensions\" and describe an iterative process in which dimensions were refined until the surveyed papers were classifiable. Appendix A's classification of 141 papers therefore re-demonstrates the design target rather than independently validating the taxonomy. The paper also claims that \"the framework is also implemented as a software system and validated to handle multiple use cases\" and that it was \"validated ... in expert interviews,\" but no protocol, results, or inter-coder agreement information is provided for either validation. This makes it impossible to assess the reliability, completeness, or objectivity of the taxonomy as a descriptive tool.","section":"§3.1 and Appendix A"},{"comment":"The classification of established feedback types uses grey and blue checkmarks to indicate that a feedback type can have different attributes across papers, but the formal definitions in Sections 3.4–3.6 are categorical (e.g., D4: |T|=1 vs. |T|>1; D8: r_fb ∈ {0,1,≻,≺} vs. N vs. R). The paper does not explain how a feedback type with multiple grey-checked attributes corresponds to a \"well-defined point\" in the nine-dimensional space, nor how the orthogonality requirement (Section 3.1) is preserved when attributes can span dimensions. This ambiguity weakens the claim that the taxonomy allows any feedback channel to sit at a well-defined point on D1–D9 and that the dimensions are mutually exclusive.","section":"§3.7 and Table 1"}],"minor_comments":[{"comment":"There are several typos in the formal definitions: \"s_i ∈, a_j ∈ A\" should read \"s_i ∈ S, a_j ∈ A\"; \"s_i ⊂ s_i ⊆ S\" in the Formal Definition of Content Level is garbled; and \"the identify function\" should be \"the identity function.\"","section":"§3.2"},{"comment":"The survey is described as covering papers from 2008 to 2024, but Appendix A and the reference list include earlier works (e.g., [75, 97, 132] from 1996–2005, [58] from 2003). Please clarify whether the 2008–2024 window applies only to the \"final survey\" keyword search and not to the earlier candidate sets.","section":"§3.1"},{"comment":"In the \"Optimizing expressiveness\" paragraph, the phrase \"open-ended, implicit or multi-modal feedback D2 options\" should be \"...feedback options related to D2\" to avoid implying that D2 itself is a set of options; similarly, \"act proactively D3\" reads awkwardly.","section":"§4.1"},{"comment":"The sentence \"Human-computer/human-robot interaction presents a huge opportunity to create novel feedback interactions\" is vague; consider specifying which interaction modalities are missing from current RLHF systems.","section":"§5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for an HCI-oriented venue such as ACM TIIS or a CHI-style venue, but the current manuscript frames itself as a definitive \"mapping out\" of the feedback space and a unifying formalization, which the evidence does not support. The scalar-bottleneck issue is not a stylistic quibble: it directly affects whether the formalization can be used as a basis for algorithm design, and the authors' own citation [113] contradicts the assumption without being reconciled. The circularity of the validation (Section 3.1) could be addressed by presenting the expert interviews and implementation validation with actual data, or by softening the exhaustiveness claim to \"taxonomy of the surveyed literature.\" The reference to the authors' own prior work [55, 113, 137] as motivation is legitimate, but the framework's dependence on [113] for the context-dependence argument should be made explicit. I recommend major revision with the expectation that the authors either provide a rigorous construction for mapping linguistic and corrective feedback to the (T, r_fb, C) triple, or explicitly scope the unification claim to feedback that is representable as scalar target evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The nine-dimension taxonomy is a real synthesis, and it earns its keep: it organizes the 141-paper corpus in a way that should give RLHF and interactive-RL researchers a shared vocabulary and a design checklist for feedback interfaces. The seven quality criteria are less novel — they read like a condensation of known issues — but mapping them onto the dimensions and deriving concrete system requirements is useful. I also credit the authors for publishing the full classification matrix in Appendix A and for being honest in Sections 6 and 7 that co-adaptation, temporal dynamics, and alternative framings are out of scope.\n\nThe soft spots are real but not fatal. Exhaustiveness is built in by design, not demonstrated: the dimensions were refined until the surveyed papers fit (Section 3.1), so Appendix A mostly re-demonstrates a design target rather than independently validating the taxonomy. The methodology mentions expert interviews (Figure 3), but no results from those interviews ever appear in the paper. That is a missing validation, and it should be either reported or explicitly dropped. The formal definitions in Section 3.2 are loose enough to be a liability: F is defined as a mapping from T to r_fb, yet T is sometimes a set of trajectories and sometimes a distribution; several formal cases contain typos (e.g., T ⊆ [(s0,a1),...]). The scalar bottleneck the stress-test flags is real: the formalism projects all feedback onto (target, scalar, context), and the authors' own earlier argument in [113] is that humans are not noisy scalar rewarders. Two semantically different messages can collapse to the same triple. That does not break the taxonomy as a design vocabulary — dimensions like D1 and D5 still capture distinctions the formalism elides — but it does mean the unification claim is overstated. The paper should either weaken the claim or engage with the information loss directly.\n\nBottom line: this deserves a serious referee. It is a good conceptual anchor for a fragmented field, and the weaknesses are addressable in revision. I would bring it to a reading group and would likely cite it for the taxonomy.","headline":"A genuinely useful taxonomy for RLHF feedback research, with an exhaustiveness claim that outruns the evidence and a scalar-reward formalization that needs to acknowledge its own information loss.","tokens_in":49524,"tokens_out":1788,"would_cite":true,"duration_ms":18996,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that all human feedback for reward-based learning—preferences, ratings, demonstrations, corrections, gaze, and language—can be classified along nine dimensions and assessed by seven quality metrics, unifying…","keywords":["reinforcement learning from human feedback","human feedback taxonomy","interactive machine learning","human-computer interaction","reward learning","feedback quality metrics","human-in-the-loop reinforcement learning","multi-type feedback"],"falsifier":"Take a concrete corrective utterance like \"Don't put the cup there, place it on the coaster, but only if the coaster is dry\" and attempt to encode it as a single (target, scalar) pair under the paper's formalism; if the resulting scalar leaves the agent unable to distinguish the dry-coaster condition from the wet-coaster one, the scalar channel demonstrably loses information the taxonomy claims to cover. Alternatively, have independent annotators classify a held-out set of feedback utterances from the surveyed papers into the nine dimensions and measure agreement; low agreement would falsify the exhaustiveness and orthogonality claims.","tokens_in":48350,"feed_emoji":"🧭","tokens_out":4374,"duration_ms":38508,"temperature":0.7,"pith_summary":"The paper sets out to answer two questions: what kinds of human feedback exist for training reinforcement learning agents, and what makes feedback good. It claims that every feedback type can be placed on nine dimensions—intent, expression form, engagement, target relation, content level, target actuality, temporal granularity, choice set size, and exclusivity—and that seven quality metrics (expressiveness, ease, definiteness, context independence, precision, unbiasedness, informativeness) govern how useful feedback is. If this is right, RLHF system builders gain a common map and vocabulary, and the field moves from preference-only feedback toward a richer design space of mixed feedback types.","feed_headline":"Nine dimensions span every human feedback type for RL","feed_subtitle":"A taxonomy unifies preferences, demonstrations, gaze, and corrections under one framework with seven quality metrics.","key_machinery":"The central object is the nine-dimensional taxonomy (D1–D9) together with the formalization of a feedback instance as a mapping $F: \\mathcal{T} \\to r_{\\text{fb}}$, produced by a translation algorithm $\\phi$ that turns raw measurements into (target, value) pairs under a context encoding. The dimensions span three groups: human-centered (intent, expression form, engagement), interface-centered (target relation, content level, target actuality), and model-centered (temporal granularity, choice set size, exclusivity). The seven quality metrics (Q1–Q7) operationalize what makes feedback good from human, interface, and model perspectives, and the derived requirements (UI.R1–R4, FP.R1–R4, RM.R1–R3) connect the taxonomy to concrete system design.","core_discovery":"The central claim is that the space of human feedback for reward-based learning is structured: every feedback utterance, whether a thumbs-up, a preference between two replies, a gaze fixation, a physical correction, or a natural-language instruction, is a point in a nine-dimensional space. The paper formalizes feedback as a mapping $F: \\mathcal{T} \\to r_{\\text{fb}}$ from a target set of trajectories to a scalar reward value, optionally conditioned on a context encoding derived from the feedback state, which decomposes into human, interface, and agent sub-states. On this foundation it builds the taxonomy, the seven quality metrics, and a set of requirements for the user interface, feedback processor, and reward model. The paper supports the claim by classifying 141 surveyed papers from 2008 to 2024 within the taxonomy.","pith_inferences":["The scalar-reward bottleneck is the framework's deepest assumption; a testable corollary is whether context-conditioned reward models that condition on the full feedback state actually recover information lost by scalar compression, something the paper motivates but does not demonstrate empirically.","The nine dimensions read naturally as an annotation schema; a natural next step the paper does not take is measuring inter-annotator agreement when independent coders classify feedback utterances, which would convert the exhaustiveness claim into a quantitative one.","The paper's own cited warning that 'humans are not Boltzmann distributions' cuts against the scalar formalism, so the framework is best read as a communication-surface map, with the reward-model semantics left as open work.","The quality metrics could be turned into a scoring rubric for RLHF interfaces, such as a checklist evaluating a system on Q1–Q7; the paper stops at defining the qualities."],"forward_implications":["RLHF systems can move beyond pairwise preferences: the framework licenses mixed feedback types, letting users choose the most natural channel (rate, correct, demonstrate, describe) at each moment.","Feedback quality becomes measurable along seven axes, enabling interface designers and reward-model trainers to compare feedback channels on the same terms.","Reward models must condition on context encodings $C$ to handle the fact that the same raw utterance means different things in different feedback states.","Querying strategies can be defined over the full nine-dimensional space, selecting not just which target to show but which feedback type to request."],"supporting_citations":[{"why":"Supplies the reward-rational (implicit) choice formalism that the paper extends to include HCI aspects.","marker":"[80]"},{"why":"Establishes the canonical pairwise-preference RLHF method that the taxonomy generalizes.","marker":"[42]"},{"why":"Represents the dominant RLHF-for-LLM usage that motivates broadening feedback beyond comparisons.","marker":"[150]"},{"why":"The paper's own prior critique of over-simplified human modeling, cited as motivation for context-aware translation.","marker":"[113]"},{"why":"Provides an alternative RLHF classification that the paper contrasts with its own nine dimensions.","marker":"[84]"},{"why":"Surveys preference-based RL, grounding the relative-feedback attribute in D4 and the binary choice set in D8.","marker":"[205]"}],"fun_headline_variants":["A 9-D map for all human feedback in RL","RL feedback: 9 axes, 7 quality metrics, one taxonomy","Bridging ML and HCI: a unified RL feedback map","One space, 9 dimensions: untangling RL feedback","9-D taxonomy: every RL feedback type, one framework"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that every kind of human feedback can be compressed into a scalar reward value (plus optional context) without losing anything essential for learning the right behavior.","fun_headline_variants_meta":{"raw":{"variants":["A 9-D map for all human feedback in RL","RL feedback: 9 axes, 7 quality metrics, one taxonomy","Bridging ML and HCI: a unified RL feedback map","One space, 9 dimensions: untangling RL feedback","9-D taxonomy: every RL feedback type, one framework"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001137,"raw_usage":{"total_tokens":4711,"prompt_tokens":925,"completion_tokens":3786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":3700}},"tokens_in":541,"tokens_out":3786,"duration_ms":25250,"temperature":1.0,"reasoning_tokens":3700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:10:57.795978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a concrete corrective utterance like \"Don't put the cup there, place it on the coaster, but only if the coaster is dry\" and attempt to encode it as a single (target, scalar) pair under the paper's formalism; if the resulting scalar leaves the agent unable to distinguish the dry-coaster condition from the wet-coaster one, the scalar channel demonstrably loses information the taxonomy claims to cover. Alternatively, have independent annotators classify a held-out set of feedback utterances from the surveyed papers into the nine dimensions and measure agreement; low agreement would falsify the exhaustiveness and orthogonality claims.","supporting_citations":[],"review_version":1}