{"id":"b3942422-02f2-4a3a-a3d6-d434b100328f","arxiv_id":"2504.13288","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A policy-gradient method for POMDPs that jointly maximizes temporal-logic task satisfaction and minimizes conditional entropy about a secret automaton state, using observable operators for gradient computation.","lead":"This paper designs a policy-gradient algorithm that lets a partially observable robot both complete a temporal-logic mission and actively gather information about a hidden event by choosing actions that affect its movement and its sensors. The method is validated on a small UAV surveillance example, but without baseline comparisons, its practical advantage over task-only planning remains unmeasured.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 10 misdefines Pθ(o,a) by dropping the initial observation likelihood; the score formula survives, so the central gradient claim is not falsified but the written derivation needs correction.","rationale":"The central mathematical claim of the paper is a policy-gradient identity for the conditional entropy of a secret automaton state in a POMDP with controllable transitions and emissions. Given an information variable Y that includes both the observation sequence and the realized action sequence, the posterior P(Z_T | Y=y) is policy-independent, so Lemma 1 follows from the log-derivative trick. The reader's weakest-assumption statement about the role of action sequences is essentially correct, and I do not find a flaw there. The most load-bearing weakness is instead Eq. 10, which is the formal definition of the trajectory probability Pθ(y) used to justify the score. As written, Eq. 10 is not a probability distribution: it drops the initial observation likelihood P(o0|a0), and the division by P(o0|a0) does not repair the observation factor for longer horizons. This is a genuine error in the manuscript. It is also a correctable presentation error rather than a falsification of the central claim, because the correct factorization immediately yields the same Proposition 3 score and the same Lemma 1 estimator. The paper should be revised to replace Eq. 10, explicitly derive Proposition 3, and clarify that the score function is independent of the normalization error. Because the reader already recommended CONDITIONAL on evaluation and presentation grounds, and the central claim survives once Eq. 10 is corrected, I do not move the verdict.","tokens_in":12337,"tokens_out":16649,"duration_ms":162530,"concrete_test":"Build a one-step toy POMDP with two states, one action, µ0 = (0.5, 0.5), E(o=1|s1,a)=0.9, E(o=1|s2,a)=0.1, and any transition kernel. Enumerate all length-1 trajectories (o0,a0). Compute Pθ(o0,a0) by the direct chain rule: πθ(a0) · Σ_s µ0(s) E(o0|s,a0). Compare with Eq. 10, which yields πθ(a0) regardless of o0. The two disagree whenever P(o0|a0) ≠ 1, confirming the equation is wrong as written. Then re-derive ∇θ log Pθ(o0,a0) from the correct factorization; it equals ∇θ log πθ(a0), verifying that Proposition 3 and the Lemma 1 estimator remain valid after the fix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 10 defines Pθ(o0:t,a0:t) = P(o0:t|a0:t) / P(o0|a0) · Π_{i=0}^t πθ(ai|o0:i−1). This is not a valid probability over (observation, action) trajectories under Definition 1. For t=0 it reduces to πθ(a0), omitting the emission probability P(o0|a0) entirely. For t>0 the denominator is an unrelated normalizer that does not cancel in the observation factor. The correct factorization, following the model in Section III, is Pθ(o0:t,a0:t) = P(o0:t|a0:t) · Π_{i=0}^t πθ(ai|o0:i−1), because the observation kernel contains no policy parameter. The proof of Lemma 1 does not itself use Eq. 10, and Proposition 3 (∇θ log Pθ(y) = Σ_t ∇θ log πθ(at|o0:t−1)) remains correct even with the typo. So the central gradient identity is not falsified. However, Eq. 10 is the written bridge between the trajectory distribution and the score function; a reader implementing it literally will compute incorrect trajectory probabilities and therefore incorrect weights for the entropy term. The manuscript should replace Eq. 10 with the correct factorization, and the derivation of Proposition 3 should be stated explicitly rather than cited from prior work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper considers a labeled POMDP in which actions influence both state transitions and observation emissions. Given an LTLf specification encoded as a DFA, the agent maximizes a weighted objective: the probability of satisfying the task minus the conditional entropy of a binary random variable Z_T indicating whether the final automaton state belongs to a secret set F_sec, where the conditioning information Y_0:T includes both observations and actions. The main technical contribution is Lemma 1, which states that the gradient of this conditional entropy with respect to policy parameters equals E_{y~P_theta}[H(Z_T|Y=y) nabla_theta log P_theta(y)], relying on the fact that H(Z_T|Y=y) is independent of the policy once the action sequence is included in y. The paper develops observable-operator formulas to compute H(Z_T|Y=y) and the task-satisfaction probability, and presents a policy-gradient algorithm. The method is demonstrated on a UAV surveillance POMDP with two possible ground-robot types and a finite-state memory policy.","tokens_in":1186,"tokens_out":2842,"duration_ms":115453,"significance":"If the derivation is corrected, the paper gives a clean, tractable policy-gradient scheme for jointly maximizing temporal-logic task satisfaction and information acquisition in POMDPs with action-dependent observations. The key identity is simple but useful, and the use of observable operators makes the entropy term computable by matrix products. The paper also explicitly handles the case where the secret set differs from the accepting set, which is a meaningful generalization beyond standard task satisfaction. The experiments, while simple, show the expected trade-off between task completion and uncertainty reduction. The main caveats are typographical errors in two central equations, not in the underlying argument; these are local and fixable.","major_comments":[{"comment":"The displayed formula for P_theta(o_0:t, a_0:t) is not a valid probability distribution. The denominator P(o_0|a_0) should not be present; the correct factorization is P_theta(o_0:t, a_0:t) = P(o_0:t|a_0:t) * prod_{i=0}^t pi_theta(a_i|o_0:i-1). As printed, for t=0 the emission probability is omitted, and for t>0 the normalization over o_0 fails. This is not used in the proof of Lemma 1, and Proposition 3 remains correct, but the manuscript must replace this equation with the correct factorization and explicitly derive Proposition 3 from it.","section":"Section IV, Eq. (10)"},{"comment":"The conditional entropy is written without the minus sign, i.e., H(Z_T|Y=y) = sum_z P(z|y) log P(z|y). This makes H non-positive and, if used in Eq. (3), would reverse the information-acquisition objective. The missing factor -1 must be restored; the definition in Section III has the correct sign.","section":"Section IV, Eq. (12)"}],"minor_comments":[{"comment":"The displayed expression is labeled as an approximation of H but is actually an approximation of nabla_theta H; the notation should be corrected for clarity.","section":"Section IV, Eq. (26)"},{"comment":"The left side uses P_theta(Z_T=1), but the quantity needed in Proposition 5 is P_theta(W_T=1); this is a typo that should be fixed.","section":"Section IV, Eq. (27)"},{"comment":"The emission function in the product POMDP is written as E(o|v), but it is used later as E(o|v,a); the action argument should be made explicit.","section":"Definition 4"},{"comment":"The gradient identity is attributed to [17], [20]; since it follows in one line from the corrected Eq. (10) by the log-derivative trick, providing the derivation in the text would make the paper self-contained.","section":"Section IV, Proposition 3"},{"comment":"The choice K=2 for the memory length is not justified; a short discussion of how the results depend on K would strengthen the evaluation.","section":"Section V"},{"comment":"There are minor typographical issues: 'UA V' should be 'UAV', and 'defaulty' should be 'default'.","section":"Abstract and Section V"}],"recommendation":"minor_revision","confidential_remarks":"The core result appears correct; the two equations flagged in the major comments are local typos that can be fixed without changing the main argument. The paper is within scope for eess.SY. I recommend minor revision after the authors correct the trajectory-probability factorization and the entropy sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece is the gradient identity in Lemma 1: for a POMDP where actions affect both transitions and emissions, the gradient of the conditional entropy of a secret automaton state given the observation-action history reduces to a weighted policy gradient. Because the conditioning includes the action sequence, the entropy term H(Z_T|Y=y) is policy-independent, so the score-function trick applies cleanly. That is a neat observation and it does generalize the authors' earlier HMM work to the full POMDP setting. The use of observable operators to compute the needed probabilities is also sensible and consistent with the prior literature they build on.\n\nThat said, the written derivation has a real flaw: Eq. 10 is malformed. It divides by P(o0|a0) for no reason and omits the initial observation likelihood. For t=0 it would claim Pθ(a0) = πθ(a0), which is just wrong as a probability over the observation-action pair. The stress-test note is right that Lemma 1's proof does not actually use Eq. 10, and Proposition 3's score gradient is correct, so the central claim survives. But a reader implementing the algorithm from the text will compute incorrect trajectory probabilities unless they fix the factorization. This must be corrected before publication.\n\nThe experimental section is the weakest part. One small graph-based UAV scenario, no baselines, no ablation of the weight α or memory length K, no code or data. The convergence plots show the algorithm works on this example, but not that it beats anything or that the trade-off curve is well characterized. These are fixable weaknesses, but they limit how much I would rely on the empirical claims.\n\nThere is also a minor citation issue: Proposition 2 is attributed to the authors' prior work, but that prior work presumably did not include action-dependent transitions. That is fine as long as the extension is explicit; here the notation suggests it is, but the malformed formula makes it hard to tell.\n\nOverall: this is a modest but legitimate step. The mathematics is mostly standard, the novelty is the combination of LTLf satisfaction with conditional-entropy minimization for a secret set of automaton states, and the gradient computation via observable operators is elegant. The paper deserves a serious referee who can verify the corrected factorization and push for better experiments. I would send it to peer review, with the expectation of major revision before acceptance.","headline":"A correct but incremental policy-gradient derivation for active perception in POMDPs with LTLf tasks, marred by a real typo in Eq. 10 and thin validation, but worth refereeing.","tokens_in":13126,"tokens_out":2289,"would_cite":true,"duration_ms":20490,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Joint control and active perception in a POMDP reduces to a weighted policy-gradient objective built on an entropy identity.","keywords":["POMDP","active perception","temporal logic","LTLf","policy gradient","conditional entropy","observable operator","information acquisition"],"falsifier":"Take a one-step POMDP with two states, two actions, and two observations in which actions affect emissions but the secret is the initial state. Enumerate all histories and compute $\\nabla_\\theta H(Z_1|Y)$ by brute-force differentiation, then compare with the right-hand side of Lemma 1 computed by the paper's observable-operator formulas. If the two disagree for a policy with a softmax parameterization, the claimed identity, and with it the gradient update, fails; the same test with a horizon of two would also check the recursion in Proposition 4.","tokens_in":12119,"feed_emoji":"📡","tokens_out":4638,"duration_ms":43189,"temperature":0.7,"pith_summary":"This paper seeks to show that a POMDP whose actions influence both state transitions and sensor emissions can be planned for two goals at once: satisfying a temporal-logic task and actively reducing uncertainty about a hidden critical event. The key move is to make the planner's information include the action history, so that the conditional entropy of the secret given that information is policy-independent at fixed histories. That makes the entropy gradient equal to an expectation of the entropy times the score function, a tractable policy-gradient update. The authors validate this on a UAV surveillance scenario where the secret is the type of a ground robot and the task changes with the type. A sympathetic reader would take the paper's claim to be that joint control and active perception is computationally accessible in this setting, not merely conceptually desirable.","feed_headline":"One gradient rule unites task success and information gain","feed_subtitle":"A POMDP policy can both satisfy temporal-logic tasks and actively reduce uncertainty about hidden events.","key_machinery":"The load-bearing object is the observable operator augmented with actions, a matrix $A_{o|a}=T_a\\,\\mathrm{diag}(O^a_{o,1},\\dots,O^a_{o,N})$ whose entries give the probability of transitioning from state $j$ to state $i$ while emitting observation $o$ under action $a$. Products of these matrices compute the probability of an observation sequence given an action sequence, and hence the posterior probability that the secret holds and the entropy $H(Z_T|Y=y)$. The paper's Lemma 1 is the identity that carries the argument: it converts the entropy gradient into an expectation over histories, and Propositions 2–4 show how to evaluate each ingredient by matrix products and the policy's softmax score. The finite-memory policy class with memory $K=2$ is the search space used in the experiments.","core_discovery":"The central claim is that, for a parametrized policy in this labeled POMDP, the gradient of the conditional entropy of the secret-state indicator $Z_T$ given the observation–action history $Y_{0:T}$ can be written exactly as $\\nabla_\\theta H(Z_T|Y;\\theta)=\\mathbb{E}_{y\\sim M_\\theta}[H(Z_T|Y=y)\\nabla_\\theta\\log P_\\theta(y)]$. Because $Y_{0:T}$ includes the actions, the conditional entropy evaluated at a fixed history does not depend on the policy parameters, so the only policy-dependent term is the probability of the history itself. The same score-function argument gives the gradient of the task-satisfaction probability $\\nabla_\\theta P_\\theta(W_T=1)=\\sum_y P(W_T=1|y)P_\\theta(y)\\nabla_\\theta\\log P_\\theta(y)$. Together these feed a gradient-descent update that minimizes a weighted sum of entropy and negative task probability, yielding a local optimum for the joint control and active-perception problem. The paper also supplies an algorithm for computing these quantities with observable-operator matrices and validates it numerically.","pith_inferences":["If the identity is as general as it appears, the same score-function structure should apply to other posterior functionals, such as mutual information between the secret and the observation stream, widening the method to richer information objectives.","The inclusion of actions in the conditioning information is the load-bearing choice; conditioning on observations alone would introduce policy-dependent posterior terms that Lemma 1 does not cover, so extending the identity to that setting would require a new derivation.","A direct testable extension would let the memory length $K$ vary and measure how much of the entropy reduction is due to memory versus active sensing, separating the value of remembering from the value of probing.","In the surveillance experiments, the entropy reduction is achieved by a policy that moves to observe the robot's goal region; this suggests the entropy term implicitly incentivizes approaching informative locations, which could be compared against a reward-based active sensing baseline."],"forward_implications":["Planners can optimize an information-theoretic objective and a temporal-logic objective in one gradient loop, so active perception emerges from the objective rather than from hand-designed sensing rewards.","The approach handles secrets that are not the task itself, such as whether an adversary was present, because the DFA is augmented with a distinguished set of secret states.","The weight $\\alpha$ gives an explicit trade-off dial: larger $\\alpha$ favors task completion, smaller favors information gain, and the same gradient machinery works across the whole range.","Because the gradient estimator is a score-function estimator, it can be combined with sample-based approximation when the history space is too large to enumerate."],"supporting_citations":[{"why":"Supplies the policy-gradient and observable-operator propositions that the paper extends from HMMs to POMDPs with controllable transitions.","marker":"[17]"},{"why":"Provides the observable operator representation used to compute probabilities of observation sequences.","marker":"[9]"},{"why":"Supplies LTLf syntax and language semantics via DFA, used in the product construction for tasks and secrets.","marker":"[6]"},{"why":"The policy gradient theorem that motivates the score-function form of the entropy gradient update.","marker":"[18]"},{"why":"Connects observable operator models to predictive state representations, supporting the matrix computation of posteriors.","marker":"[19]"}],"fun_headline_variants":["Gradient identity unites task success and information gain","One score-function gradient for control and active perception","Shared gradient rule for task satisfaction and entropy reduction","Policy gradient ties task completion to hidden-state info"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gradient identity depends on conditioning on the full action–observation history; if the planner's information were limited to observations alone, the posterior entropy would depend on the policy and the paper's simple score-function gradient would no longer hold.","fun_headline_variants_meta":{"raw":{"variants":["Gradient identity unites task success and information gain","One score-function gradient for control and active perception","Shared gradient rule for task satisfaction and entropy reduction","Policy gradient ties task completion to hidden-state info"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000837,"raw_usage":{"total_tokens":3638,"prompt_tokens":924,"completion_tokens":2714,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2654}},"tokens_in":540,"tokens_out":2714,"duration_ms":19604,"temperature":1.0,"reasoning_tokens":2654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:12:50.749547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a one-step POMDP with two states, two actions, and two observations in which actions affect emissions but the secret is the initial state. Enumerate all histories and compute $\\nabla_\\theta H(Z_1|Y)$ by brute-force differentiation, then compare with the right-hand side of Lemma 1 computed by the paper's observable-operator formulas. If the two disagree for a policy with a softmax parameterization, the claimed identity, and with it the gradient update, fails; the same test with a horizon of two would also check the recursion in Proposition 4.","supporting_citations":[{"cited_title":"Active perception with initial- state uncertainty: A policy gradient method,","cited_arxiv_id":null,"evidence_quote":"Supplies the policy-gradient and observable-operator propositions that the paper extends from HMMs to POMDPs with controllable transitions."},{"cited_title":"Observable Operator Models for Discrete Stochastic Time Series,","cited_arxiv_id":null,"evidence_quote":"Provides the observable operator representation used to compute probabilities of observation sequences."},{"cited_title":"Linear temporal logic and linear dynamic logic on finite traces,","cited_arxiv_id":null,"evidence_quote":"Supplies LTLf syntax and language semantics via DFA, used in the product construction for tasks and secrets."},{"cited_title":"Policy gradi- ent methods for reinforcement learning with function approximation,","cited_arxiv_id":null,"evidence_quote":"The policy gradient theorem that motivates the score-function form of the entropy gradient update."},{"cited_title":"Links between multiplicity automata, observable operator models and predictive state representations – a unified learning framework,","cited_arxiv_id":null,"evidence_quote":"Connects observable operator models to predictive state representations, supporting the matrix computation of posteriors."}],"review_version":1}