Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging Agents

Engin Erzin; Nusrah Hussain; T. Metin Sezgin; Yucel Yemez

REVIEW 4 major objections 5 minor 28 references

Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an offline-trained recurrent Q-agent can time laughter backchannels to produce more estimated engagement than an imitation-trained agent.

desk verdict A plausible but unproven application of batch RL to backchannel generation; the central engagement claim is confounded by the reward counting the agent's own laughs. read the letter →

arxiv 1908.02037 v1 pith:O274N4CX submitted 2019-08-06 cs.AI cs.LG

Nusrah Hussain , Engin Erzin , T. Metin Sezgin , Yucel Yemez This is my paper · ORCID

classification cs.AIcs.LG

keywords batchreinforcementlearningbackchannelgenerationhuman-robotinteractionengagementpartiallyobservableMarkovdecisionprocessdeeprecurrentQ-learningoff-policypolicyevaluationlaughter

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a robot can learn when to produce laughter as a backchannel without interacting with humans during training, by treating a recorded dyadic conversation dataset as batch reinforcement learning data. The problem is framed as a partially observable Markov decision process: states are one-second audio feature summaries, the action is whether to laugh, and the reward is a scalar engagement measure based on connection events such as mutual gaze and backchannels. The central claim is that a Q-network with a recurrent long short-term memory layer learns a policy whose expected engagement, estimated by off-policy importance sampling, is higher than the engagement of an agent trained by supervised imitation of human laughs. This matters because tutoring and counseling robots need engaging backchannel behavior but cannot be trained through days of trial-and-error interaction with people.

What carries the argument

The load-bearing machinery is a deep recurrent Q-network: an MLP whose first fully connected layer is replaced by a long short-term memory (LSTM) layer, trained on truncated sequences of 80 time steps (two seconds) sampled from each dialogue. It approximates $Q(o,a|\theta)$ for a POMDP in which observations are 209-dimensional summaries of MFCC and prosody features; the recurrence narrows the gap between $Q(o,a)$ and $Q(s,a)$. The evaluation machinery is the step-wise weighted importance-sampling estimator of Equation (4), with behavior-policy probabilities estimated by approximate nearest neighbor and deterministic greedy policies assigned 95% probability.

What would settle it

Compute the effective sample size or weight variance of the step-wise importance-sampling estimator under a laugh-heavy policy: if the weights collapse onto a handful of trajectories, or if clipping the weights erases the 30.7 advantage, the engagement claim is an artifact of the estimator. A companion test is a human-subject study comparing engagement ratings for the recurrent policy against an imitation policy.

Watch

Extended reading notes

Core claim

The central discovery is that recurrent structure in the approximate value function substantially improves batch Q-learning for backchannels. In off-policy evaluation, the fully connected LSTM network reaches a maximum estimated engagement of 30.7, compared with 24.8 for a plain multilayer perceptron and 21.47 for the average engagement in the batch data itself. The authors read this as evidence that an RL agent is expected to generate more engagement than an imitation-learning agent, because the recurrent value function can accumulate history beyond the one-second state window and better approximates the true Q-values in a partially observable environment.

Load-bearing premise

Everything hinges on the assumption that the importance-sampling estimate stays trustworthy even though the learned policies laugh far more than the 1.5% laugh rate in the recorded data, so the estimator must extrapolate behavior-policy probabilities over states the dataset rarely visits.

Editorial extensions

If this is right

If the central claim is right, a social robot can be trained for backchannel behavior without days of online human interaction, since the policy is learned from a static dataset.
The recurrent value function should generalize to other partially observable interaction tasks where states are short audio or visual windows.
The same batch-RL pipeline can be applied to other backchannel types such as nods, smiles, and verbal acknowledgments.
The off-policy estimates suggest the learned policy is not merely imitating the recorded behavior policy, since it exceeds the dataset's average engagement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not pursue is testing whether the engagement advantage survives on other dyadic datasets, since the reward and behavior-policy estimates are tied to the actors in the recorded dialogues.
One consequence left implicit is that the 30.7 estimate could partly reward frequent laughter; clipping the laugh rate and re-estimating would show how much of the gain comes from timing versus quantity.
If the importance weights are degenerate at high laugh rates, the result would be an estimator artifact; reporting effective sample size would settle this.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this is a genuine new combination—batch RL with recurrent Q-networks trained on a human-human dyadic dataset (IEMOCAP) to select laugh backchannels—but the quantitative evidence for the main claim is weak. The paper's own OPE numbers don't establish that the RL policy beats imitation learning, for a fixable but real reason: the agent's laugh is both the action and a counted connection event in the reward.

What's good: the problem is well motivated, the batch-RL framing is sensible for HRI, and the state/action/reward definitions are concrete. They do leave-one-subject-out cross-validation, and the LSTM-vs-MLP comparison is a useful result for partially observable state in this setting. The Bellman residual curves show stable training. The writing is clear and the related work is well placed.

The soft spots are, in order of importance. (1) Reward self-counting. In Section 4.1, backchannels including laughs are connection events, and the reward is pace = 1/MTBCE. So a laugh by the behavior-policy actor directly increases the reward. In the step-WIS OPE of Eq. (4), timesteps where the behavior policy happens to laugh carry both a higher r_t and, under a laugh-heavy evaluation policy, a much larger importance weight. The estimated advantage of the RL policy is therefore partly measuring the policy's laugh rate rather than user engagement. Removing the agent's own laughs from the CE count is a necessary fix. (2) No variance or confidence intervals on the OPE estimates. The reported maxima (30.7 vs 24.8) are selected from curves swept over laugh thresholds on the same OPE, so there is selection bias. This is minor relative to (1) but still serious. (3) Behavior policy estimation and coverage are not analyzed; the learned policies laugh far more than the dataset's 1.5% rate, which should raise red flags about importance-weight degeneracy.

The Bellman residual comparison is fine as a training diagnostic, but it says nothing about how the policy would perform with a human user.

Net: the idea is worth exploring and the paper deserves a serious referee—a good reviewer could ask for a reward that excludes the agent's own laughs, confidence intervals, and ideally a small human-subject pilot. As it stands, I wouldn't cite it as evidence for the engagement claim, and I'd be skeptical of the abstract's statement until the OPE is cleaned up. If you're working on social-robot backchannel generation, it's worth a skim for the formulation.

Referee Report

4 major / 5 minor

Summary. The paper proposes a batch reinforcement learning method for backchannel generation in human-robot interaction, using laughs as the target backchannel and the IEMOCAP dyadic corpus as a fixed batch of off-policy trajectories. States are formed from summary statistics of MFCC and prosody features over a one-second window; the action is a binary laugh/no-laugh decision; and the reward is a scalar engagement measure (pace = 1/MTBCE) derived from connection events, including backchannels. The authors train two Q-network variants, an MLP and a fully connected LSTM, and compare them against a supervised classification baseline. Evaluation relies on Bellman residuals and step-wise weighted importance sampling (step-WIS) off-policy evaluation. The paper reports that the FC-LSTM reaches an estimated value of 30.7 versus 24.8 for the MLP and 21.47 for the dataset average, and concludes that the RL agents are expected to produce more engagement than an imitation-learned agent.

Significance. If the central claim were firmly established, the paper would make a useful contribution by showing that offline RL on a pre-existing human-human interaction corpus can improve over supervised backchannel generation, and by demonstrating that recurrent value function approximation helps under partial observability. The subject-independent LOSO split and the use of a real human-human dataset as a batch-RL benchmark are strengths, and the study addresses a problem of practical interest for social robotics. However, as presented, the evidence for the main claim is not conclusive: the reward definition conflates the agent's own laugh action with engagement, and the off-policy evaluation lacks variance and coverage analysis. The significance of the contribution therefore depends on whether these issues can be resolved; the current manuscript is better read as a proof-of-concept than as a validated comparison.

major comments (4)

[Section 4.1 (Reward) and Section 6.2, Eq. (4)] The reward is pace = 1/MTBCE, computed from connection events, and backchannels 'that include laughs, smiles, nods and head-shakes' are explicitly counted as connection events. Since the agent's action is exactly laughter, every laugh by the behavior-policy actor contributes positively to the reward at that timestep. In the step-WIS estimator, rewards r_t at laugh timesteps are high by construction, and under a laugh-heavy evaluation policy those same timesteps receive large importance weights. The OPE advantage in Fig. 4 therefore does not separate 'the policy produces laughter' from 'the policy produces engagement.' This confound affects the comparison with the supervised baseline independently of the coverage issue. The authors should recompute rewards excluding the agent's own backchannels from the connection-event count, or use an independent annotation of user engagement that does not include the action being learned.
[Section 6.2, Eq. (4) and Fig. 4] The step-WIS estimate is reported without any variance or confidence intervals and without coverage diagnostics. The behavior policy has a laugh rate of about 1.5%, while Fig. 4 sweeps the fraction of laughs from 0 to 1; the evaluation policies therefore assign large probability to actions that are rare or absent under the estimated behavior policy, so the importance ratios π/π_b can be degenerate or unbounded. The behavior policy is estimated from the same data via approximate nearest neighbor, but no calibration or comparison to the empirical laugh rate is reported. In addition, the maximum values (30.7 vs 24.8) are selected by sweeping a threshold on the same OPE curve, which is an in-sample selection that inflates the reported advantage. The paper needs effective sample sizes, variance estimates, importance-weight clipping or smoothing, and held-out threshold selection to support the headline comparison.
[Section 6.1, Eq. (3)] The claim that a smaller Bellman residual shows the learned policy is 'closer to optimal' is not justified as stated. Bellman residuals over the batch can be minimized by functions that are not optimal Q-functions, especially with nonlinear function approximation and off-policy data; the residual also depends on the state distribution in the batch. Thus the lower LSTM residual in Fig. 2 should be presented only as a fitting diagnostic, not as evidence of better policy optimality. This weakens the conclusion in Section 7 that the training is successful according to 'various objective metrics.'
[Section 6.2 and Section 5] It is unclear whether the OPE in Fig. 4 is computed on the held-out test fold or on the same batch used for training. The text says the batch is split 4:1 with leave-one-subject-out and that results are subject independent, but the OPE description does not state which trajectories enter Eq. (4). If the OPE uses the training batch, the comparison is in-sample; if it uses the test fold, the paper should say so explicitly and report per-fold estimates. The same ambiguity applies to the Bellman residual in Eq. (3), which is described as being computed 'over the entire batch data B.'

minor comments (5)

[Section 4.1 (heading)] The heading 'Batch-RL Formulaion' contains a typo; it should read 'Batch-RL Formulation.'
[Section 5 (Q Networks discussion)] The sentence 'since it did perform as well' should read 'since it did not perform as well'; otherwise the sentence contradicts the exclusion of the second LSTM variant from the reported results.
[Fig. 4] The x-axis label 'Fraction of actions with laughs' should specify whether this is the fraction of actions taken by the evaluation policy and how the thresholds mentioned in Section 6.2 map to that fraction.
[Section 5 and Fig. 4] The supervised baseline is described as producing probabilities via softmax and then being thresholded, but it is not stated whether the thresholding procedure is identical to that used for the RL policies, nor whether the 'Dataset' line in Fig. 4 is the average per-step reward of the behavior policy or the discounted return; these details should be clarified.
[Section 5 (Experiments)] The paper does not report the number of random restarts or seeds, initialization details, or the range of hyperparameters tried for the sequence length L and learning rate; adding this information would improve reproducibility of the numerical comparisons.

Circularity Check

2 steps flagged · score 6.0 of 10

Engagement advantage is partly forced by a reward definition that counts the agent's own laugh as a connection event; the headline OPE numbers are also maxima over a swept threshold.

self definitional [Section 4.1 (Batch-RL Formulation), used in Section 6.2 OPE with Eq. (4)]
"Action: Agent’s action is a binary variable, indicating the absence or presence of laugh. Laughs of the actor described as the behavioral policy are labeled at a rate of 40 Hz. ... we use the connection events (CE) (1) mutual facial gaze, (2) adjacency pair and (3) backchannels (that include laughs, smiles, nods and head-shakes) to quantify engagement."

The engagement reward is defined to count backchannels, including laughs, while the agent's action is exactly the presence or absence of a laugh. Therefore a policy that laughs more raises the pace reward component by construction, independent of any measured change in the user. In the step-WIS estimator of Eq. (4), logged rewards r_t from the behavior policy's laugh events are reweighted by importance ratios; laugh-heavy evaluation policies assign large weights to exactly those laugh timesteps, so the estimated value is inflated. The reported 30.7 versus 24.8 advantage is thus partly a consequence of the reward definition counting the agent's own action as engagement, not an independently measured engagement gain.
fitted input called prediction [Section 6.2, Figure 4]
"The OPE is performed at different threshold values (or amount of generated laughs) and similar to before a 95% probability is assigned to the suggested action. Fig. 4 shows the values estimated by WIS at different fraction of laughs. The fully connected LSTM clearly outperforms MLP, producing a maximum value of 30.7 versus 24.8, whereas the batch data consists of an average value of 21.47."

The threshold controlling the laugh rate is not fixed in advance; it is swept, and the reported headline numbers are the maxima of the resulting OPE curves. Presenting the best point on the swept curve as the method's result is a post hoc selection on the evaluation metric, so the claimed 'clearly outperforms' comparison is partly a selected maximum rather than a fixed-policy prediction. This fits the pattern of a fitted parameter being renamed as a predicted result.

full rationale

The central result is not a first-principles derivation; it is an empirical RL comparison. However, the key claim that the RL agents 'are expected to produce more engagement than an agent trained from imitation learning' depends on two reductions. First, engagement is operationalized through connection events, and backchannels (including laughs) are counted as connection events; the action space is exactly laugh/no-laugh. A policy that produces more laughs directly raises the pace reward, so the estimated value difference in Eq. (4) partly measures the policy's own laugh frequency, not an independent external engagement signal. The step-WIS estimator reweights logged rewards; laugh timesteps carry both high reward (because a laugh is a connection event) and high importance weight under laugh-heavy evaluation policies, so the comparison is inflated by construction. Second, the reported 'maximum value of 30.7 versus 24.8' is obtained after sweeping threshold values over the fraction of laughs; taking the maximum of the OPE curve is a post hoc selection, so the headline numbers are selected rather than predicted. These issues are partial: the reward also includes mutual facial gaze and adjacency pair, and the threshold sweep does not erase the LSTM-versus-MLP structural comparison, so the paper is not wholly circular. No load-bearing self-citation appears: the author citations ([1], [22]) support only background assumptions about laughter and feature construction and are not what forces the result. Because the central engagement claim is in part defined in terms of the manipulated action and the announced advantage is a swept maximum, the circularity score is 6.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim depends on several assumptions the paper does not independently validate: the reward proxy, the MDP/POMDP tuple extraction, the OPE validity, and the interpretation of Bellman residuals. Counted free parameters include manually chosen windows, discount factor, sequence length, network widths, and post hoc laugh thresholds. No invented entities are introduced.

free parameters (7)

discount factor gamma = 0.99
Hand-chosen in Section 5; affects all Q-values and the weighted OPE return.
pace window = 15 seconds
Reward engagement pace is computed over a 15-second window; not derived, chosen by authors in Section 4.1.
state window / frame rate = 1 second at 25 ms / 40 Hz
Speech feature state uses the past one second at 40 Hz (Section 4.1); no tuning or justification is given.
LSTM sequence length L = 80 steps (2 seconds)
Selected because 'L=80 to be stable while improving the results' (Section 5); this is a validation-driven choice, not a prediction.
OPE laugh threshold = Selected from plot; maximum occurs at moderate laugh fraction
In Section 6.2 thresholds define deterministic policies, and the reported maxima (30.7, 24.8) are points on curves swept over this threshold; this is post hoc tuning.
greedy action probability in OPE = 95%
Authors assign 95% probability to the action suggested by the deterministic policy before computing importance weights (Section 6.2).
neural network widths = 209-100-25 for MLP and 209-LSTM100-25 for FC-LSTM
Hidden layer sizes are chosen without justification in Section 4.2.

assumptions (6)

domain assumption IEMOCAP dyadic conversations can be re-labeled as MDP/POMDP trajectories from a behavior policy, with one actor as agent and the other as environment.
Section 4.1 constructs tuples <st,at,rt,st+1>; this treats human actors as if their laughs were actions selected by a policy.
domain assumption The pace engagement metric from connection events is a valid scalar reward for backchannel generation.
Section 4.1 uses pace = 1/MTBCE over a 15-second window as reward; no validation that increasing this proxy corresponds to user engagement.
domain assumption A one-second audio feature window plus LSTM memory is a sufficient observation design for the true interaction state.
Sections 1 and 4.1 define states from one second of speech; partial observability is acknowledged and addressed with recurrency, but sufficiency is assumed.
domain assumption The behavior policy can be estimated from the batch data and step-WIS importance sampling is valid for comparing policies.
Section 6.2 estimates the behavior policy with approximate nearest neighbor and applies Eq. (4); no coverage or variance checks are reported.
ad hoc to paper A small Bellman residual over the batch indicates closeness to an optimal policy.
Section 6.1 interprets lower Bellman residuals as 'better optimality'; in batch RL with function approximation this need not hold, and residuals are reported on the training batch.
standard math Standard Q-learning and Bellman control equation are applicable to this problem.
Section 2.2 uses standard Q-learning and the Bellman control equation as background.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging Agents." pith.science (2026). https://pith.science/paper/O274N4CX

@misc{pith2026190802037,
  author       = {Pith},
  title        = {Pith review of: Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O274N4CX}},
  note         = {Machine review of arXiv:1908.02037}
}

read the original abstract

The ability to generate appropriate verbal and non-verbal backchannels by an agent during human-robot interaction greatly enhances the interaction experience. Backchannels are particularly important in applications like tutoring and counseling, which require constant attention and engagement of the user. We present here a method for training a robot for backchannel generation during a human-robot interaction within the reinforcement learning (RL) framework, with the goal of maintaining high engagement level. Since online learning by interaction with a human is highly time-consuming and impractical, we take advantage of the recorded human-to-human dataset and approach our problem as a batch reinforcement learning problem. The dataset is utilized as a batch data acquired by some behavior policy. We perform experiments with laughs as a backchannel and train an agent with value-based techniques. In particular, we demonstrate the effectiveness of recurrent layers in the approximate value function for this problem, that boosts the performance in partially observable environments. With off-policy policy evaluation, it is shown that the RL agents are expected to produce more engagement than an agent trained from imitation learning.

Figures

Figures reproduced from arXiv: 1908.02037 by the authors.

**Figure 2.** Bellman residual vs training samples 0 50 100 150 200 250 No. of Samples (in millions) 10 15 20 25 30 35 Average Q-value MLP FC-LSTM (6sec) FC-LSTM (2sec) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 4.** Off-Policy Policy Evaluation 7 Conclusion and Future Work We demonstrated in this work batch reinforcement learning methods for training an agent to learn to produce backchannels with the objective of maximizing the user’s engagement.State modeling proved challenging for this problem and in general, may be described as a partially observable Markov decision process. Using audio features for state modeling, we succe… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

[1]

Analysis of engagement and user experience with a laughter responsive social robot,

B. B. Türker, Z. Buçinca, E. Erzin, Y . Yemez, and M. Sezgin, “Analysis of engagement and user experience with a laughter responsive social robot,” inProc. 18th Annu. Conf. Int. Speech Commun. Assoc , 2017, pp. 844–848

work page 2017
[2]

Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent,

B. Inden, Z. Malisz, P. Wagner, and I. Wachsmuth, “Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent,” inProceedings of the 15th ACM on International conference on multimodal interaction. ACM, 2013, pp. 181–188

work page 2013
[3]

Deﬁnitions of engagement in human-agent interaction,

N. Glas and C. Pelachaud, “Deﬁnitions of engagement in human-agent interaction,” in 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) . IEEE, 2015, pp. 944–949

work page 2015
[4]

Poggi, Mind, hands, face and body: a goal and belief view of multimodal communication

I. Poggi, Mind, hands, face and body: a goal and belief view of multimodal communication . Weidler, 2007

work page 2007
[5]

Deep recurrent q-learning for partially observable mdps,

M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” CoRR, abs/1507.06527, vol. 7, no. 1, 2015

arXiv 2015
[6]

Q-learning,

C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992

1992
[7]

R. S. Sutton and A. G. Barto, Introduction to reinforcement learning. MIT press Cambridge, 1998, vol. 135

work page 1998
[8]

Batch reinforcement learning,

S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” inReinforcement learning. Springer, 2012, pp. 45–73

work page 2012

Show all 28 references

[9]

Human-level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,”Nature, vol. 518, no. 7540, p. 529, 2015

2015
[10]

Tree-based batch mode reinforcement learning,

D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode reinforcement learning,” Journal of Machine Learning Research, vol. 6, no. Apr, pp. 503–556, 2005

2005
[11]

Neural ﬁtted q iteration–ﬁrst experiences with a data efﬁcient neural reinforcement learning method,

M. Riedmiller, “Neural ﬁtted q iteration–ﬁrst experiences with a data efﬁcient neural reinforcement learning method,” in European Conference on Machine Learning. Springer, 2005, pp. 317–328. 7

2005
[12]

Robot gains social intelligence through multimodal deep reinforcement learning,

A. H. Qureshi, Y . Nakamura, Y . Yoshikawa, and H. Ishiguro, “Robot gains social intelligence through multimodal deep reinforcement learning,” in2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids). IEEE, 2016, pp. 745–751

2016
[13]

Show, attend and interact: Perceivable human-robot social interaction through neural attention q-network,

——, “Show, attend and interact: Perceivable human-robot social interaction through neural attention q-network,” in 2017 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2017, pp. 1639–1645

2017
[14]

Robot behavior adaptation for human-robot interaction based on policy gradient reinforcement learning,

N. Mitsunaga, C. Smith, T. Kanda, H. Ishiguro, and N. Hagita, “Robot behavior adaptation for human-robot interaction based on policy gradient reinforcement learning,”Journal of the Robotics Society of Japan , vol. 24, no. 7, pp. 820–829, 2006

2006
[15]

Neural network based reinforcement learning for audio–visual gaze control in human–robot interaction,

S. Lathuilière, B. Massé, P. Mesejo, and R. Horaud, “Neural network based reinforcement learning for audio–visual gaze control in human–robot interaction,” Pattern Recognition Letters, vol. 118, pp. 61–71, 2019

2019
[16]

Fostering user engagement in face-to-face human-agent interactions: a survey,

C. Clavel, A. Cafaro, S. Campano, and C. Pelachaud, “Fostering user engagement in face-to-face human-agent interactions: a survey,” inToward Robotic Socially Believable Behaving Systems-V olume II. Springer, 2016, pp. 93–120

2016
[17]

Generating robot/agent backchannels during a storytelling experiment,

S. Al Moubayed, M. Baklouti, M. Chetouani, T. Dutoit, A. Mahdhaoui, J.-C. Martin, S. Ondas, C. Pelachaud, J. Urbain, and M. Yilmaz, “Generating robot/agent backchannels during a storytelling experiment,” in2009 IEEE International Conference on Robotics and Automation . IEEE, 2...

2009
[18]

Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction,

C. Liu, C. T. Ishi, H. Ishiguro, and N. Hagita, “Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction,” in 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI) . IEEE, 2012, pp. 285–292

2012
[19]

Data-driven model of nonverbal behavior for socially assistive human-robot interactions,

H. Admoni and B. Scassellati, “Data-driven model of nonverbal behavior for socially assistive human-robot interactions,” in Proceedings of the 16th international conference on multimodal interaction . ACM, 2014, pp. 196–199

2014
[20]

Recognizing engagement in human-robot interaction,

C. Rich, B. Ponsler, A. Holroyd, and C. L. Sidner, “Recognizing engagement in human-robot interaction,” in Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on . IEEE, 2010, pp. 375–382

2010
[21]

Iemocap: Interactive emotional dyadic motion capture database,

C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, no. 4, p. 335, 2008

2008
[22]

Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gestures,

E. Bozkurt, E. Erzin, and Y . Yemez, “Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gestures,” Speech Communication, vol. 85, pp. 29–42, December 2016

2016
[23]

Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information,

A. Metallinou, A. Katsamanis, and S. Narayanan, “Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information,”Image and Vision Computing, vol. 31, no. 2, pp. 137–152, 2013

2013
[24]

Residual algorithms: Reinforcement learning with function approximation,

L. Baird, “Residual algorithms: Reinforcement learning with function approximation,” in Machine Learning Proceedings 1995. Elsevier, 1995, pp. 30–37

1995
[25]

Data-efﬁcient off-policy policy evaluation for reinforcement learning,

P. Thomas and E. Brunskill, “Data-efﬁcient off-policy policy evaluation for reinforcement learning,” inInterna- tional Conference on Machine Learning , 2016, pp. 2139–2148

2016
[26]

Importance sampling for fair policy selection

S. Doroudi, P. S. Thomas, and E. Brunskill, “Importance sampling for fair policy selection.”Grantee Submission, 2017

2017
[27]

Behaviour policy estimation in off-policy policy evaluation: Calibration matters,

A. Raghu, O. Gottesman, Y . Liu, M. Komorowski, A. Faisal, F. Doshi-Velez, and E. Brunskill, “Behaviour policy estimation in off-policy policy evaluation: Calibration matters,”arXiv preprint arXiv:1807.01066, 2018

2018 arXiv
[28]

Fast nearest neighbor search through sparse random projections and voting,

V . Hyvönen, T. Pitkänen, S. Tasoulis, E. Jääsaari, R. Tuomainen, L. Wang, J. Corander, and T. Roos, “Fast nearest neighbor search through sparse random projections and voting,” in Big Data (Big Data), 2016 IEEE International Conference on. IEEE, 2016, pp. 881–888. 8

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Analysis of engagement and user experience with a laughter responsive social robot,

B. B. Türker, Z. Buçinca, E. Erzin, Y . Yemez, and M. Sezgin, “Analysis of engagement and user experience with a laughter responsive social robot,” inProc. 18th Annu. Conf. Int. Speech Commun. Assoc , 2017, pp. 844–848

work page 2017

[2] [2]

Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent,

B. Inden, Z. Malisz, P. Wagner, and I. Wachsmuth, “Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent,” inProceedings of the 15th ACM on International conference on multimodal interaction. ACM, 2013, pp. 181–188

work page 2013

[3] [3]

Deﬁnitions of engagement in human-agent interaction,

N. Glas and C. Pelachaud, “Deﬁnitions of engagement in human-agent interaction,” in 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) . IEEE, 2015, pp. 944–949

work page 2015

[4] [4]

Poggi, Mind, hands, face and body: a goal and belief view of multimodal communication

I. Poggi, Mind, hands, face and body: a goal and belief view of multimodal communication . Weidler, 2007

work page 2007

[5] [5]

Deep recurrent q-learning for partially observable mdps,

M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” CoRR, abs/1507.06527, vol. 7, no. 1, 2015

arXiv 2015

[6] [6]

Q-learning,

C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992

1992

[7] [7]

R. S. Sutton and A. G. Barto, Introduction to reinforcement learning. MIT press Cambridge, 1998, vol. 135

work page 1998

[8] [8]

Batch reinforcement learning,

S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” inReinforcement learning. Springer, 2012, pp. 45–73

work page 2012