pith. sign in

arxiv: 2603.04959 · v2 · submitted 2026-03-05 · 💻 cs.HC

Beyond Advocacy: A Design Space for Replication-Related Studies

Yiheng Liang , Kim Marriott , Helen C. Purchase This is my paper

Pith reviewed 2026-05-15 16:55 UTC · model grok-4.3

classification 💻 cs.HC
keywords replicationdesign spaceexperimental designHCIvisualizationpairwise comparisonreplication studiesresearch methods
0
0 comments X

The pith

Replication experimental design can be framed as a pairwise comparison across four practical dimensions each with three levels.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that calls to replicate studies remain largely at the level of advocacy because there is no shared language for the concrete design choices that distinguish one replication from another. It offers a design space that models each replication as a set of explicit comparisons between the original study and the new one. These comparisons are organized along four dimensions that capture what stays the same and what is altered. The framework is intended to serve both as a retrospective tool for classifying existing replications and as a planning aid for designing new ones. A reader would care because it replaces vague debates about “direct” versus “conceptual” replication with a practical grid that makes the scope of any given replication visible and debatable.

Core claim

The central claim is that replication design is usefully treated as a pairwise comparison problem and can be represented by a four-dimensional design space in which each dimension is defined by three comparison levels. This space allows replication studies to be identified, categorized, compared and analyzed systematically, supporting both retrospective characterization of past work and prospective planning of future studies.

What carries the argument

The four-dimensional design space that represents replication as pairwise comparisons between original and replication studies at three discrete levels per dimension.

If this is right

  • Researchers gain a shared vocabulary for specifying exactly which aspects of an original study they intend to hold constant.
  • Existing replication papers can be systematically classified and gaps in coverage become visible.
  • Planning a new replication becomes a matter of choosing a point in the design space rather than an unstructured set of decisions.
  • Reviewers and readers can more easily assess whether a claimed replication actually tests the intended claim.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same grid could be applied to replication debates in fields outside HCI and visualization to test whether the four dimensions travel.
  • Standard reporting templates for replication studies could incorporate explicit positions on each dimension.
  • Meta-analyses of replication success rates might be refined by grouping studies that occupy similar positions in the space.
  • Tool builders could create simple interfaces that let authors record their design choices along the four dimensions at submission time.

Load-bearing premise

That the four chosen dimensions and their three comparison levels are enough to describe the important design decisions in any replication study without significant omissions.

What would settle it

A concrete replication study whose key decisions about what to keep or change cannot be placed on any of the four dimensions without forcing or leaving out major aspects of the design.

read the original abstract

The importance of replication is often discussed and advocated -- not only in the domains of visualization and HCI, but in all scientific areas. When replicating a study, design decisions need to be made with regards which aspects of the original study will remain the same and which will be altered. We present a supporting multi-dimensional design space framework within which such decisions can be identified, categorized, compared and analyzed. The framework treats replication experimental design as a pairwise comparison problem, and represents the design by four practical dimensions defined by three comparison levels. The design space is therefore a framework that can be used for both retrospective characterization and prospective planning. We provide worked examples, and relate our framework to other attempts at describing the scope of replication studies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes a multi-dimensional design space framework for replication-related studies in HCI and visualization. It frames replication experimental design as a pairwise comparison problem between an original study and its replication, represented by four practical dimensions each defined at three comparison levels. The framework is positioned as a tool for both retrospective characterization of existing replications and prospective planning of new ones. The manuscript supplies worked examples and relates the framework to prior attempts at scoping replication studies.

Significance. If the four dimensions and three levels prove jointly exhaustive, the framework would provide a structured, analyzable alternative to purely advocative discussions of replication, enabling clearer categorization and comparison of design choices. This could improve planning and evaluation of replications in visualization and HCI, particularly by making trade-offs explicit, though its value hinges on demonstrated coverage and usability beyond the provided examples.

major comments (2)
  1. [Framework definition section] The central claim that the four dimensions at three comparison levels comprehensively capture all relevant design decisions (abstract and framework definition) rests on worked examples without a systematic mapping from an enumerated set of replication decisions drawn from the literature. This leaves the exhaustiveness untested and risks omissions such as adjustments to statistical power, preregistration status, or hardware/platform constraints that may not align with the stated levels.
  2. [Worked examples section] Section on worked examples: The examples illustrate application of the framework but do not include an audit or coverage check against a broader sample of replication studies, which is required to substantiate that the dimensions can identify, categorize, compare, and analyze decisions without significant gaps.
minor comments (2)
  1. The abstract would benefit from naming the four dimensions explicitly to allow readers to grasp the framework's structure immediately.
  2. Notation for the three comparison levels could be standardized with a table or diagram for easier reference across sections.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our design space framework for replication-related studies. We address each major comment below, clarifying our approach and outlining targeted revisions to strengthen the manuscript's justification without overstating the current evidence.

read point-by-point responses

  1. Referee: [Framework definition section] The central claim that the four dimensions at three comparison levels comprehensively capture all relevant design decisions (abstract and framework definition) rests on worked examples without a systematic mapping from an enumerated set of replication decisions drawn from the literature. This leaves the exhaustiveness untested and risks omissions such as adjustments to statistical power, preregistration status, or hardware/platform constraints that may not align with the stated levels.

    Authors: The four dimensions were derived from a review of replication literature in HCI and visualization, focusing on recurring pairwise design decisions between original and replication studies. We did not conduct a formal systematic enumeration of every possible decision across all papers, which is a limitation we acknowledge. However, we can demonstrate coverage for the cited examples: statistical power adjustments map to the 'analysis' dimension at the modified level; preregistration status is captured under the 'procedure' dimension as a protocol change; and hardware/platform constraints align with the 'environment' dimension. We will revise the framework definition section to include an explicit mapping table of common replication decisions from the literature to our dimensions and levels, addressing potential gaps directly. revision: partial

  2. Referee: [Worked examples section] Section on worked examples: The examples illustrate application of the framework but do not include an audit or coverage check against a broader sample of replication studies, which is required to substantiate that the dimensions can identify, categorize, compare, and analyze decisions without significant gaps.

    Authors: The worked examples were selected to span different replication types (direct, conceptual, and extension) to illustrate practical use for both retrospective and prospective purposes. We agree that a broader audit would provide stronger substantiation of coverage. We will add a new subsection (or appendix) auditing an additional set of 10-15 replication studies drawn from the HCI and visualization literature, explicitly mapping their design decisions to the four dimensions and three levels while noting any edge cases or how they are accommodated. revision: yes

Circularity Check

0 steps flagged

No circularity: standalone conceptual framework with no derivations or self-referential reductions

full rationale

The manuscript proposes a multi-dimensional design space for replication-related studies as a conceptual tool, defining four practical dimensions at three comparison levels directly by author construction to support retrospective characterization and prospective planning. No equations, fitted parameters, predictions, or mathematical derivations appear in the provided text. The framework is presented as a new organizing structure with worked examples and relations to prior work, but without any load-bearing step that reduces by construction to its own inputs, self-citations, or fitted data. The central claim of utility for identifying and categorizing design decisions rests on the explicit definition of the dimensions and levels rather than any circular equivalence or imported uniqueness theorem. This is a self-contained conceptual contribution with no circular elements.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on standard domain assumptions about experimental design in HCI rather than new postulates or fitted parameters.

axioms (1)
  • domain assumption Replication studies require explicit decisions on which elements to hold constant versus alter relative to the original.
    Invoked as the foundational premise for treating replication design as a pairwise comparison problem.

pith-pipeline@v0.9.0 · 5414 in / 1026 out tokens · 36512 ms · 2026-05-15T16:55:14.178919+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.