REVIEW 3 major objections 4 minor 16 references
Championing Research Through Design in HRI
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Human-robot interaction should adopt Research through Design so the field can ask what is worth designing, not just how to design it.
desk verdict A clear, honest position paper advocating Research through Design for HRI; the proposal is reasonable, but the key claim about how RtD will get adopted is asserted, not demonstrated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is Research through Design (RtD), defined as inquiry in which designers generate new knowledge through the process of designing and making. A key operational tool is Speed Dating with User Enactments, where participants experience brief, 'brought-to-life' scenarios using actors, prototypes, and props, then reflect on their needs and values. This method allows exploration of interactions that do not yet exist without requiring controlled experiments or internal validity. The paper also invokes speculative design as a related exploratory approach for questioning assumptions about robots in domestic life.
What would settle it
A review of HRI conference submissions over the next several years showing that exploratory RtD papers are consistently rejected for lacking controlled user studies would contradict the paper's expectation that RtD can enter HRI through CHI.
Extended reading notes
Core claim
The central claim is that the HRI community can and should accept exploratory design research, specifically Research through Design, as a valid contribution alongside controlled user studies. On the paper's account, RtD generates knowledge through the process of designing and making artifacts, allowing researchers to investigate 'wicked' problems where the design space is open. In HRI, where design papers typically create and test a robot task, adopting RtD would expand the boundaries of the field to include questions about when designing robots is the right thing to do. The paper supports this by presenting examples such as the Drift Table and speculative robots, and by describing the authors' use of Speed Dating with User Enactments to study re-embodiment of social presence across robot bodies.
Load-bearing premise
The argument rests on the premise that the HRI community will eventually accept exploratory design work as a legitimate contribution, even though the paper itself notes that the method is unstructured and lacks internal validity.
Editorial extensions
If this is right
- HRI design contributions could include exploratory knowledge about what to design, not only evaluations of existing robot functions.
- RtD could help the field avoid design fixation, as seen in the Whisper example, by re-framing assumptions about user behavior.
- The authors' re-embodiment study could be shared with the HRI audience once RtD is accepted, addressing questions of when social presence should move between bodies.
- Publishing RtD work at CHI could create a cross-community group that gradually introduces RtD into HRI.
- HRI designers could identify which RtD methods need adaptation for robot-specific questions.
Reading between the lines
- If RtD becomes accepted, HRI might also embrace other critical design practices like speculative design and value-sensitive design, broadening what counts as a design contribution.
- RtD could be used to ask normative questions about robot autonomy and social roles, such as whether certain tasks should be automated at all, which current task-performance evaluation cannot address.
- A testable extension: measure whether HRI papers that include RtD-style explorations lead to different research questions or robot concepts than papers that start from a predefined task.
- The transfer from CHI to HRI may require explicit review criteria for RtD, since journal and conference reviewers in HRI often expect internal validity.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This workshop paper argues that the HRI community should adopt Research through Design (RtD) methods from CHI in order to expand the accepted forms of design contribution beyond task-oriented evaluation. It characterizes RtD as exploratory design that asks "what is the right thing to design," gives several CHI examples (Whisper, the Drift Table, Odom et al.'s virtual-possessions work), mentions Auger's speculative design robots as a related HRI-adjacent example, and reports the authors' own use of Speed Dating with User Enactments to explore robot re-embodiment. The paper then identifies the main barrier—RtD is unstructured and does not satisfy standard "user study" review criteria—and concludes with a proposal that HRI designers publish RtD work on HRI topics in CHI venues, creating a cross-community group that will allow RtD to enter HRI more naturally.
Significance. The paper is a position statement rather than an empirical study, and as such its contribution is to articulate a plausible gap between CHI and HRI design-research norms and to propose a bridging strategy. The argument is internally coherent, and the illustrative examples are real and clearly relevant to the CHI side of the discussion. The authors also deserve credit for explicitly acknowledging the central difficulty—that RtD is unstructured by definition and will look methodologically weak when reviewed as a user study. If the central normative claim is correct, the paper points to a concrete shift in how HRI might evaluate design research, moving beyond task performance toward exploratory knowledge about robots' roles and contexts. The main weakness is that the empirical premises about HRI's current design publications and about how CHI publishing norms might transfer to HRI are asserted rather than supported; the authors themselves concede the mechanism is still to be examined.
major comments (3)
- [Section 4, final paragraph] The central recommendation—that publishing RtD on HRI topics in CHI will "create a growing group of researchers... and allow RtD to enter HRI more naturally"—is load-bearing but asserted without evidence. Section 3 explicitly concedes that the authors "would like to examine... how to encourage HRI researchers to accept RtD," which admits that the transfer mechanism is untested. Because the practical value of the paper's proposal depends entirely on this cross-community influence, the manuscript should either reframe this as an explicit hypothesis with a description of how it could be evaluated, or supply evidence that CHI publication patterns have previously shifted acceptance criteria in HRI.
- [Section 1, third paragraph] The premise that "the majority of design papers published in HRI create and test a particular robot task or function" is stated without any citation or systematic review of the HRI literature. This empirical claim is the basis for identifying the gap that the paper asks to close. As written, the claim is unsupported, and the argumentative weight is too large for an anecdotal generalization. The authors should at least cite an existing review of HRI design research or narrowly qualify the claim as their personal observation.
- [Section 2] The examples (Whisper, the Drift Table, Odom et al., Auger) establish that RtD and speculative design have produced valued knowledge in CHI and adjacent communities, but they do not demonstrate that HRI's current design practices are inferior or that HRI would benefit from adopting RtD. To support "the HRI community could greatly benefit," the paper would need a structured analysis of HRI design contributions or an explicit labeling of the benefit claim as a hypothesis to be debated in the workshop. Without one of these, the gap between the evidence and the conclusion remains open.
minor comments (4)
- [Section 1] The sentence "With some notable exceptions, the majority of design papers published in HRI create and test a particular robot task or function" has a subject-verb agreement issue: "papers... create and test" is correct, but the phrase "in HRI creates" is grammatically awkward; revise for clarity.
- [Page 1, header/DOI] The DOI in the footer is a placeholder ("https://doi.org/10.1145/nnnnnnn.nnnnnnn") and should be replaced or removed before any archival submission.
- [References, [14]] Reference [14] (Overgoor and Funk, "IdleBot: exploring the design of serendipitous artifacts") lists the venue as the ACM SIGACCESS Conference on Computers and Accessibility, which appears inconsistent with the topic of serendipitous artifacts in domestic settings; please verify the correct publication venue and update the entry.
- [Section 3] The authors' own Speed Dating study is used as an illustrative case, but its placement in the argument should make explicit that it is an example of method use rather than evidence that HRI reviewers would accept such work; a single sentence clarifying this would avoid the appearance of circular support.
Circularity Check
No derivational circularity: the paper is a position/advocacy argument, and its self-citations are illustrative rather than load-bearing.
full rationale
This workshop paper makes no quantitative derivation, fits no parameters, and produces no empirical prediction that could reduce to its own inputs. Its central claim is that HRI could benefit from adopting Research through Design (RtD), supported by historical examples from CHI (Whisper, Drift Table, Odom et al.) and by the authors' own RtD work on re-embodiment. The self-citations, e.g., [10], [11], [12], [13], and [15], are present and numerous, but they are not used as a proof step that forces the conclusion. They document prior RtD practice and provide concrete projects; the conclusion that HRI should accept RtD is argued from those examples and from the stated gap in HRI's accepted design methods, not derived from the citations themselves. The paper's own Section 3 limitation statement is candid: RtD is 'not structured or controlled,' and reviewing it from a 'user study' perspective would reveal 'plenty of methodology gaps.' This is a genuine limitation concerning the transferability of CHI acceptance norms to HRI, and it is an empirical claim about community culture, not a circularity. The weakest link is the unsupported sociological prediction that publishing RtD on HRI topics in CHI will 'allow RtD to enter HRI more naturally,' but unsupported prediction is a correctness or evidence risk, not a circular self-derivation. There is no equation or definition that makes the output equal to the input, and no fitted value is renamed as a prediction. Accordingly, the circularity score is minimal and reflects only the presence of minor, non-load-bearing self-citations.
Assumptions & free parameters
assumptions (4)
- domain assumption RtD produces valid research knowledge through designing and making.
- domain assumption Design fixation is a real risk in HRI and exploratory design can overcome it.
- domain assumption Internal validity and systematic coverage are not required when exploring wicked problems.
- ad hoc to paper CHI's history of RtD acceptance can be replayed in HRI by publishing in CHI.
Cite this review
Pith. "Pith review of Championing Research Through Design in HRI." pith.science (2026). https://pith.science/paper/DW6SNK5F
@misc{pith2026190807572,
author = {Pith},
title = {Pith review of: Championing Research Through Design in HRI},
year = {2026},
howpublished = {\url{https://pith.science/paper/DW6SNK5F}},
note = {Machine review of arXiv:1908.07572}
}
read the original abstract
One of the challenges in conducting research on the intersection of the CHI and Human-Robot Interaction (HRI) communities is in addressing the gap of acceptable design research methods between the two. While HRI is focused on interaction with robots and includes design research in its scope, the community is not as accustomed to exploratory design methods as the CHI community. This workshop paper argues for bringing exploratory design, and specifically Research through Design (RtD) methods that have been established in CHI for the past decade to the foreground of HRI. RtD can enable design researchers in the field of HRI to conduct exploratory design work that asks what is the right thing to design and share it within the community.
Figures
Reference graph
Works this paper leans on
-
[1]
James Auger. 2014. Living with robots: A speculative design approach. Journal of Human-Robot Interaction 3, 1 (2014), 20–42
work page 2014
-
[2]
Richard Buchanan. 1992. Wicked problems in design thinking. Design issues 8, 2 (1992), 5–21
work page 1992
-
[3]
Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design . Morgan Kaufmann
work page 2010
-
[4]
Evangelia G Chrysikou and Robert W Weisberg. 2005. Following the wrong footsteps: fixation effects of pictorial examples in a design problem-solving task. Journal of Experimental Psychology: Learning, Memory, and Cognition 31, 5 (2005), 1134
work page 2005
-
[5]
Scott Davidoff, Min Kyung Lee, Anind K Dey, and John Zimmerman
-
[6]
Brian R Duffy, Gregory MP O’Hare, Alan N Martin, John F Bradley, and Bianca Schon. 2003. Agent chameleons: Agent minds and bodies. In Proceedings 11th IEEE International Workshop on Program Compre- hension. IEEE, 118–125
work page 2003
-
[7]
Anthony Dunne and Fiona Raby. 2013. Speculative everything: design, fiction, and social dreaming . MIT press
work page 2013
-
[8]
Masaaki Fukumoto and Yoshinobu Tonomura. 1999. Whisper: a wrist- watch style wearable handset. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems . ACM, 112–119
work page 1999
Show all 16 references
-
[9]
William W Gaver, John Bowers, Andrew Boucher, Hans Gellerson, Sarah Pennington, Albrecht Schmidt, Anthony Steed, Nicholas Villars, and Brendan Walker. 2004. The drift table: designing for ludic en- gagement. In CHI’04 extended abstracts on Human factors in computing systems. A...
2004
-
[10]
Ilpo Koskinen, John Zimmerman, Thomas Binder, Johan Redstrom, and Stephan Wensveen. 2011. Design research through practice: From the lab, field, and showroom . Elsevier
2011
-
[11]
Michal Luria, Samantha Reig, Xiang Zhi Tan, Aaron Steinfeld, Jodi Forl- izzi, and John Zimmerman. 2019. Re-Embodiment and Co-Embodiment: Exploration of social presence for robots and conversational agents. In Proceedings of the 2019 on Designing Interactive Systems Conference ...
2019
-
[12]
William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user en- actments. In Proceedings of the Designing Interactive Systems Conference. CHI ’19 Workshops, May 2019, Glasgow, UK Luria et al. ACM, 338–347
2012
-
[13]
William Odom, John Zimmerman, and Jodi Forlizzi. 2011. Teenagers and their virtual possessions: design opportunities and issues. In Pro- ceedings of the SIGCHI conference on Human Factors in computing sys- tems. ACM, 1491–1500
2011
-
[14]
Caroline Overgoor and Mathias Funk. 2018. IdleBot: exploring the design of serendipitous artifacts. InProceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 105– 110
2018
-
[15]
John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 493–502
2007
-
[2007]
In Int’l Conference on Ubiquitous Computing
Rapidly exploring application design through speed dating. In Int’l Conference on Ubiquitous Computing . Springer, 429–446
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.