Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Active-MoSH claims that learning from iterative soft-hard bound adjustments gets decision makers to their ideal Pareto-optimal point in fewer interactions than standard feedback, with stronger confidence.

desk verdict A genuinely interactive extension of MoSH with an honest user study, but the T-MoSH advantage is partly injected by the simulation protocol, so the central trust/efficiency claim needs a cleaner test before it lands. read the letter →

arxiv 2506.21887 v2 pith:WXU44SFN submitted 2025-06-27 cs.AI cs.LG

Edward Chen , Sang T. Truong , Natalie Dullerud , Sanmi Koyejo , Carlos Guestrin This is my paper · ORCID
classification cs.AIcs.LG
keywords multi-objectiveoptimizationpreferencelearningsoft-hardboundsinteractivedecision-makingBayesianinferenceactivequerysamplingsensitivityanalysisuserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

High-stakes multi-objective choices—a cancer treatment plan, a structural design, a cover image—force a decision maker to balance competing goals with expensive evaluations. This paper proposes Active-MoSH, which turns the way experts already talk about goals—aspirational soft bounds and non-negotiable hard bounds per objective—into a Bayesian preference-learning signal. Each adjustment of a bound is read as an implicit ranking of the displayed candidate points, and the system actively selects the next small query set while a global sensitivity-analysis component flags potentially overlooked high-value regions. If the framework works as claimed, decision makers reach their ideal Pareto-optimal point in fewer interactions, with more confidence, than with pairwise, ranking, or uniform feedback; the paper offers simulations, a brachytherapy case, and an image-selection user study as evidence.

What carries the argument

The central object is the soft-hard utility function ($u_\alpha$), a bounded utility that rewards objective values near the aspirational soft bound and forbids values beyond the non-negotiable hard bound, combined across objectives by a scalarization $s_\lambda(u_\alpha(x))$. The load-bearing mechanism is the interpretation of a bound adjustment as an implicit ranking of the displayed points, scored by a Plackett-Luce likelihood, which turns expert constraint-talk into a full preference signal. Active sampling then maximizes the expected SHF utility ratio over posterior draws of $\lambda$ and $\alpha$, while T-MoSH maximizes expected improvement with the perturbed hard bound slightly relaxed to identify potentially overlooked, high-value regions.

What would settle it

Record what users actually do: give a participant the Active-MoSH interface, have them adjust a hard or soft bound, then separately ask them to rank the points shown. If the stated rankings disagree with distance-to-the-new-bound ordering across users and iterations, the Plackett-Luce likelihood in Eq. (5) is misspecified and the posterior updates are not tracking real preferences.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that the familiar expert act of restating targets and limits—'keep tumor coverage above 95% if possible, never below 90%, and keep bladder dose under 601 cGy'—is enough to drive a preference-learning loop. Active-MoSH treats each bound adjustment as an implicit ranking of the K displayed Pareto points, with points closest to the newly adjusted bound ranked highest, and updates posteriors over the hidden preference vector $\lambda$ and hidden ideal bounds $\alpha$ through a Plackett-Luce likelihood. It then samples new queries by maximizing the expected SHF utility ratio, using Gaussian-process surrogates and random scalarizations from the posteriors, and sparsifies them to a small set. The global component T-MoSH maximizes expected improvement under a slightly relaxed hard bound to surface high-value points outside the current region. The paper asserts that this combined loop finds a decision maker's ideal point with fewer interaction units than pairwise, ranking, or uniform feedback, and that the sensitivity-based global view increases confidence; the user study is presented as evidence of both.

Load-bearing premise

The ranking interpretation is the load-bearing premise: when a decision maker moves a soft or hard bound, the framework assumes the displayed points are ranked by distance to the new bound, and if real users adjust bounds for global reasons rather than to signal which displayed points they like, the posteriors can converge to the wrong preferences.

Editorial extensions

If this is right

  • With the same number of interaction units, Active-MoSH reaches a higher SHF utility ratio than pairwise, full ranking, partial ranking, and uniform feedback on the Branin-Currin, Four Bar Truss, and brachytherapy benchmarks.
  • The ablation results show that dropping the Plackett-Luce preference update or replacing active sampling with random sampling degrades convergence, so the paper's efficiency claim depends on both components.
  • Active-T-MoSH, which adds T-MoSH sensitivity analysis, generally converges at least as fast and with lower variance than Active-MoSH alone.
  • In the user study, participants reached higher utility at the second feedback iteration with Active-T-MoSH than with pairwise and full-ranking feedback, and they rated it more expressive and more trustworthy, while also rating it more mentally demanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that T-MoSH's expected-improvement screen can double as a stopping rule: when no candidate outside the current bounds has positive expected improvement, the system has a defensible answer to 'have I missed anything?'.
  • A testable extension is to learn the ranking interpretation itself, letting bound adjustments and explicit rankings coexist so the likelihood can be corrected if users adjust bounds for global rather than local reasons.
  • The interaction-unit accounting could be validated directly by measuring task-completion time or physiological effort, which would sharpen the efficiency claim beyond the paper's assumption-heavy unit assignments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Active-MoSH, an interactive framework for multi-objective optimization in which a decision-maker (DM) iteratively adjusts soft and hard bounds on objectives. The local component models DM preferences and bounds with Bayesian posteriors, interprets bound adjustments as implicit rankings via a Plackett-Luce likelihood (Eq. 5), and uses an active sampling strategy to present small, Pareto-optimal query sets. The global component T-MoSH uses multi-objective sensitivity analysis to propose points that may indicate overlooked improvements, aiming to build DM trust. The authors evaluate Active-MoSH against pairwise, ranking, and uniform feedback baselines in simulations on synthetic and real-world problems (including brachytherapy) and in a user study on AI-generated image selection. They claim improved convergence efficiency and trust, measured by SHF Utility Ratio and an Iteration Stop Efficiency metric.

Significance. The problem is important: interactive multi-objective decision-making with expressive, dual-level preferences is relevant to high-stakes applications, and the paper is among the first to formalize iterative soft/hard bound adjustments probabilistically. The framework is principled, and the sparse query selection inherits a submodularity guarantee (Theorem 1). The paper also attempts a human-subject validation, which is commendable. However, the main validation is undermined by a simulation protocol that injects the T-MoSH advantage (A.4.1) and by a user study whose primary trust metric was not significant. As currently presented, the evidence does not yet establish the claimed efficiency and trust benefits.

major comments (4)
  1. [Appendix A.4.1 (simulation setup)] The simulation protocol directly injects the benefit of T-MoSH: when y* is outside the bounds and T-MoSH promotes an improved point, the simulated DM is assumed to gain enhanced confidence and increase the magnitude of the bound adjustment by 5%. Since this magnitude boost is the only systematic difference between the Active-MoSH and Active-T-MoSH arms in the synthetic experiments, the higher SHF Utility Ratios for Active-T-MoSH in Figures 3, 4, 5, and 8 cannot be attributed to the information T-MoSH provides. The authors should either re-run the comparison without this assumption or justify the 5% boost with human behavioral data.
  2. [Section 3.2 and Appendix A.4.1] The Section 3.2 feedback interpretation is load-bearing: the posterior updates in Eq. (3) rely on a Plackett-Luce likelihood over rankings induced by proximity to the adjusted bound. The simulated DM in A.4.1 is implemented to choose the point in Ym closest to y* as the reference and to rank points by distance to the new bound, which is exactly the same rule as Eq. (5). Consequently, the synthetic experiments cannot independently validate the ranking interpretation; they only show the method converges when the DM behaves as modeled. The paper needs an external test of this assumption, for example a user study that directly measures how users rank displayed points after a bound adjustment, or a comparison with alternative feedback models on real user data.
  3. [Section 6.3, Eq. (10)] The primary behavioral metric for trust, Iteration Stop Efficiency (ISE), did not reach statistical significance against any baseline (adj. p = 0.15 vs. pairwise, 0.18 vs. partial ranking, 0.29 vs. full ranking). The significant result on the Likert trust question is a self-report that may be confounded with perceived expressiveness or novelty, and the user study did not include an Active-MoSH (without T-MoSH) condition, so the specific contribution of T-MoSH to trust is not isolated. The abstract's claim that the framework 'enhance[s] DM trust' is therefore not supported by the behavioral evidence.
  4. [Section 2.2, Eq. (2) and Eq. (5); Appendix A.4.1] The evaluation metric SHF Utility Ratio (Eq. 2) and the feedback likelihood (Eq. 5) are both defined via the same SHF utility function, and the simulation ground truth y* is sampled to lie in the high-utility regions of the chosen soft and hard bounds (A.4.1). This creates a circular evaluation: the simulation rewards methods that align with the SHF model that the method itself assumes. To establish external validity, the authors should include at least one evaluation that does not depend on SHF utilities, such as distance to the known Pareto point under an independent utility model, or behavioral outcome measures from the user study.
minor comments (5)
  1. [Section 5.2 and Figure 3] The x-axis of Figures 3--5 is not explicitly described in the text or captions; please add axis labels and state whether the horizontal axis is feedback units or iterations.
  2. [Appendix A.2.1] The Metropolis-Hastings implementation uses only 20 burn-in steps for the posterior over the preference vector; please provide diagnostics or a sensitivity analysis for this choice.
  3. [References] There are several incomplete references, e.g., 'Ziebart et al., Shaikh et al., 2024' in the related work section; please correct the citation entries.
  4. [Section 6.1] The user study measures self-reported trust with a single Likert item; consider using a validated trust scale to improve reliability.
  5. [Figure 5] When each feedback instance is assigned a single unit, the comparison at '10 units' means different numbers of feedback instances per method; please clarify this interpretation in the caption.

Circularity Check

2 steps flagged · score 6.0 of 10

The simulated T-MoSH advantage is partly built in: the simulated DM is assumed to make 5% larger bound adjustments when T-MoSH is active, and the Active-MoSH simulator uses the same proximity ranking the model assumes.

  1. other [Appendix A.4.1, Simulation Setup]
    "When y∗ is outside the bounds and our proposed method T-MoSH promotes an improved point, we assume the DM has enhanced confidence and increases the magnitude with which they modify the soft or hard bound for that iteration. We assume this increase to be 5%."

    In the simulation, Active-MoSH and Active-T-MoSH differ by this 5% adjustment-magnitude boost (plus the sensitivity-analysis samples). Larger bound adjustments move the simulated DM's bounds further toward the known y∗, so the Active-T-MoSH arm receives larger corrective steps by construction. The convergence advantage attributed to T-MoSH in Figures 3-5 is therefore partially an input assumption about the very trust/efficiency benefit T-MoSH is claimed to provide, rather than an independent measurement of the value of T-MoSH's sensitivity information. The user study's primary trust metric ISE did not reach significance (adj. p = 0.15 vs. pairwise), so this simulated boost is load-bearing for the trust/efficiency claim.

  2. self definitional [Section 3.2 / Appendix A.2.1 / Appendix A.4.1]
    "In practice, the ranking of points in Ym is determined by their Euclidean distance to the bound being modified at the end of iteration m, following the intuition described in Section 3.2."

    The framework's likelihood (Eq. 5) models bound adjustments as an implicit Plackett-Luce ranking by proximity to the adjusted bound through the SHF utility. The simulator generates Active-MoSH feedback by selecting the displayed point closest to the known y∗ and adjusting the bound toward it, i.e., by the same proximity-to-bound rule. Thus the synthetic experiments cannot independently validate the Section 3.2 feedback interpretation; they test the algorithm under its own assumption. The user study provides partial independent evidence, but the simulation-based efficiency comparisons are self-confirming for this load-bearing modeling choice.

full rationale

The paper's core probabilistic interactive framework is not circular in itself: the SHF utility, MoSH-Dense, and MoSH-Sparse are imported from prior work (Chen et al., 2024) with stated proofs and external algorithmic content, and the active-learning posterior updates are new. However, the central simulated evidence for T-MoSH's trust and efficiency benefit is contaminated by an explicit modeling assumption: the simulated DM is assumed to make 5% larger bound adjustments whenever T-MoSH promotes an improved point, directly injecting the effect the simulation is meant to demonstrate. Additionally, the simulated Active-MoSH feedback is generated using the same proximity-based ranking that the model assumes in its Plackett-Luce likelihood, so the synthetic comparisons validate the model's internal consistency rather than the realism of the feedback interpretation. The user study supplies some independent human evidence, and the Likert trust ratings were significant, but the primary quantitative trust metric (ISE) was not significant, and the user study did not isolate T-MoSH's contribution. Overall, the derivation chain is not fully circular, but one key simulated 'prediction' reduces to an assumed input, warranting a score of 6.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the SHF utility model, the Plackett-Luce ranking interpretation of bound adjustments, and several hand-set simulation parameters. No new physical entities are introduced. The most consequential assumption is that a user's bound adjustment maps cleanly to a ranking of displayed points, which is also encoded in the simulated DM.

free parameters (6)
  • SHF utility shape parameter beta = 0.25
    Controls the slope of the SHF in the soft region; set in A.1.1 following Chen et al. It affects all utilities, rankings, and the evaluation metric.
  • SHF saturation factor zeta = 2.0
    Sets alpha_tau = alpha_H + zeta(alpha_S - alpha_H) in A.1.1; taken from Chen et al. and affects utility saturation.
  • Simulated DM default adjustment magnitudes = soft 0.3, hard 0.4
    Used in A.4.1 to simulate Active-MoSH feedback; hand-set and directly shapes simulation results.
  • T-MoSH confidence boost = 5% increase
    In A.4.1, when T-MoSH promotes a point, the simulated DM is assumed to adjust bounds 5% more, directly affecting the reported Active-T-MoSH gains.
  • Interaction unit weights = Active-MoSH 2, ranking k, pairwise 1
    Section 5.2 assigns cognitive costs for comparing feedback mechanisms; the authors provide a uniform-unit alternative in Section 5.6.
  • UCB exploration scale beta_t = sqrt(0.125 * log(2t+1))
    Used in the acquisition function in Section 3.3; adopted from Paria et al. and affects exploration during active sampling.
assumptions (6)
  • domain assumption The DM has a fixed hidden ideal set of soft and hard bounds and scalarization weights that define an ideal point y*.
    Section 2.2 Problem Definition; the entire learning target is defined this way.
  • domain assumption Adjusting a soft/hard bound induces a ranking of displayed points by distance to the new bound, modeled by Plackett-Luce over SHF utilities.
    Section 3.2 and Eq 5; if real DM feedback does not follow this, the likelihood and posterior are misspecified.
  • domain assumption Posterior distributions over bounds remain Gaussian and are updated with Gaussian likelihoods.
    Section 3.1, Eq 4; used for bound posterior updates and sampling.
  • standard math The SHF Utility Ratio F_lambda is submodular, so MoSH-Sparse gives near-optimal sparse queries via the Krause et al. theorem.
    Section 3.3, Theorem 1; relies on the proof from Chen et al. 2024.
  • domain assumption Random scalarization with weighted Chebyshev scalarization and GP-UCB acquisition finds Pareto-optimal points robust to posterior uncertainty.
    Section 3.3 and A.4.1; standard but unvalidated for non-convex frontiers, such as the brachytherapy case.
  • domain assumption Expected improvement over the GP posterior identifies overlooked high-value points for building trust.
    Section 4, Eq 8-9; assumes that maximizing EI outside current bounds reveals trust-relevant alternatives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds." pith.science (2026). https://pith.science/paper/WXU44SFN

@misc{pith2026250621887,
  author       = {Pith},
  title        = {Pith review of: Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WXU44SFN}},
  note         = {Machine review of arXiv:2506.21887}
}
read the original abstract

High-stakes decision-making involves navigating multiple competing objectives with expensive evaluations. For instance, in brachytherapy, clinicians must balance maximizing tumor coverage (e.g., an aspirational target or soft bound of >95% coverage) against strict organ dose limits (e.g., a non-negotiable hard bound of <601cGy to the bladder). Selecting Pareto-optimal solutions that match implicit preferences is challenging, as exhaustive Pareto frontier exploration is computationally and cognitively prohibitive, necessitating interactive frameworks to guide users. While decision-makers (DMs) often possess domain knowledge to narrow the search via such soft-hard bounds, current methods often lack systematic approaches to iteratively refine these multi-faceted preference structures. Furthermore, DMs often require confidence that they have not overlooked superior alternatives, a paramount necessity in high-stakes scenarios. We present Active-MoSH, an interactive local-global framework designed for this process. Its local component integrates probabilistic preference learning with an active sampling strategy to adaptively refine Pareto subsets while minimizing cognitive burden. To bolster decision confidence, Active-MoSH's global component, C-MoSH, leverages multi-objective sensitivity analysis to identify potentially overlooked, high-value points beyond immediate feedback. We demonstrate Active-MoSH's performance benefits through diverse synthetic and real-world applications. A high-stakes case study with real cervical cancer brachytherapy treatment plans and an image selection user study further validate our hypotheses regarding the framework's ability to improve convergence, enhance DM confidence, and provide expressive preference articulation.

Figures

Figures reproduced from arXiv: 2506.21887 by the authors.

Figure 1
Figure 1. Example feedback iteration with Active-MoSH for cervical cancer brachytherapy, balancing [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Soft and Hard Bounds Feedback Interpretation. Adjustment of soft and hard bounds impose [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Evaluation results with 10 interaction units for the Branin-Currin synthetic function, Four [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (42 more)
Figure 4
Figure 4. Figure 4: Evaluation results at 10 interaction units for the DTLZ2 three-objective synthetic function, [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Evaluation results with 10 units for the Branin-Currin synthetic function, Four Bar Truss [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Ablation results for our proposed preferences modeling component, where we use opera [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Ablation results for our proposed active query sampling component, where we leveraged a [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Evaluation results at 10 interaction units for the Branin-Currin synthetic function, where the [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Sample set of query images (Albert Einstein) presented as part of the educational magazine [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Results from our image generation selection-based user study. Top row (left to right): SHF [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Example of a normalized soft-hard bounded utility function. The dashed vertical bars [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Distribution of Likert-scale survey responses for our third end-of-study question for the [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Input prompt passed into GPT-4o-mini to generate the expanded prompt for an image with [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Input prompt passed into GPT-4o-mini to evaluate the AI-generated images in terms of [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Input prompt passed into GPT-4o-mini to evaluate the AI-generated images in terms of [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Input prompt passed into GPT-4o-mini to evaluate the AI-generated images in terms of [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: MTurk User Study Instructions: Introduction, Study Overview, and Task Summary. [PITH_FULL_IMAGE:figures/full_fig_p032_23.png]
Figure 24
Figure 24. Figure 24: MTurk User Study Instructions: Description of Key Image Attributes and the Realism [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]
Figure 25
Figure 25. Figure 25: MTurk User Study Instructions: Description of Full Ranking, Partial Ranking, and [PITH_FULL_IMAGE:figures/full_fig_p034_25.png]
Figure 26
Figure 26. Figure 26: MTurk User Study Instructions: Description of the Soft & Hard Bounds feedback [PITH_FULL_IMAGE:figures/full_fig_p035_26.png]
Figure 27
Figure 27. Figure 27: MTurk User Study Instructions: Step-by-step study process. [PITH_FULL_IMAGE:figures/full_fig_p036_27.png]
Figure 28
Figure 28. Figure 28: MTurk User Study Instructions: Tips for selection, technical requirements, and concluding [PITH_FULL_IMAGE:figures/full_fig_p036_28.png]
Figure 29
Figure 29. Figure 29: Survey questions presented to MTurk participants after their interaction with each feedback [PITH_FULL_IMAGE:figures/full_fig_p037_29.png]
Figure 30
Figure 30. Figure 30: MTurk User Study Screening Quiz: Questions 1-3, focusing on understanding the Soft [PITH_FULL_IMAGE:figures/full_fig_p038_30.png]
Figure 31
Figure 31. Figure 31: MTurk User Study Screening Quiz: Questions 4-5, assessing understanding of the Partial [PITH_FULL_IMAGE:figures/full_fig_p039_31.png]
Figure 32
Figure 32. Figure 32: Login interface for the Interactive Multi-Objective Decision Making Tool used in the user [PITH_FULL_IMAGE:figures/full_fig_p039_32.png]
Figure 33
Figure 33. Figure 33: Instructional material presented to user study participants, illustrating the tradeoff between [PITH_FULL_IMAGE:figures/full_fig_p040_33.png]
Figure 34
Figure 34. Figure 34: Additional instructional examples illustrating the realism and color vividness tradeoff, [PITH_FULL_IMAGE:figures/full_fig_p040_34.png]
Figure 35
Figure 35. Figure 35: User interface for the Full Ranking feedback mechanism. Participants are presented [PITH_FULL_IMAGE:figures/full_fig_p040_35.png]
Figure 36
Figure 36. Figure 36: Interface displaying “Choices Seen So Far” during a task instance. This screen allows [PITH_FULL_IMAGE:figures/full_fig_p041_36.png]
Figure 37
Figure 37. Figure 37: User interface for the Partial Ranking feedback mechanism. Participants are tasked with [PITH_FULL_IMAGE:figures/full_fig_p041_37.png]
Figure 38
Figure 38. Figure 38: User interface for the Pairwise Comparison feedback mechanism. Participants are shown [PITH_FULL_IMAGE:figures/full_fig_p042_38.png]
Figure 39
Figure 39. Figure 39: User interface for the Soft-Hard Bounds feedback mechanism (Active-T-MoSH). Partici [PITH_FULL_IMAGE:figures/full_fig_p042_39.png]
Figure 40
Figure 40. Figure 40: User study task description for selecting an Albert Einstein cover image for "Historical [PITH_FULL_IMAGE:figures/full_fig_p043_40.png]
Figure 41
Figure 41. Figure 41: User study task description for selecting an Albert Einstein cover image for "Mind Blast [PITH_FULL_IMAGE:figures/full_fig_p044_41.png]
Figure 42
Figure 42. Figure 42: User study task description for selecting an Albert Einstein cover image for "Science [PITH_FULL_IMAGE:figures/full_fig_p045_42.png]
Figure 43
Figure 43. Figure 43: User study task description for selecting a Barack Obama cover image for "Presidential [PITH_FULL_IMAGE:figures/full_fig_p046_43.png]
Figure 44
Figure 44. Figure 44: User study task description for selecting a Barack Obama cover image for "American [PITH_FULL_IMAGE:figures/full_fig_p047_44.png]
Figure 45
Figure 45. Figure 45: User study task description for selecting a Barack Obama cover image for "Future Leaders: [PITH_FULL_IMAGE:figures/full_fig_p048_45.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 39 canonical work pages

  1. [1]

    Instructions. We provided each MTurk participant with a detailed set of instructions covering the tradeoff between the objectives, the different feedback mechanisms and how to use them, a walkthrough of the entire workflow, and other details. We provide the entire set of instructions in Figures 23-28

    work page

  2. [2]

    Screening quiz. We gave interested MTurk participants a five-question screening quiz on the concepts which were explained in the set of instructions, mainly for ensuring understanding 24 of the feedback types. We provide our screening quiz questions in Figures 30 and 31. The screening quiz interface was developed using Streamlit str [2018] and the data wa...

    work page 2018

  3. [3]

    Each Human Intelligence Task (HIT) consisted of four different task instances

    User study tool. Each Human Intelligence Task (HIT) consisted of four different task instances. For each task instance, the MTurk participants were provided with the task description, along with an interface to provide their feedback on a set of query images at each iteration. The MTurk participants were instructed to provide feedback on the set of query ...

    work page 2018

  4. [4]

    Use of complementary colors - pairing colors from opposite sides of the color wheel

    work page

  5. [5]

    discord - whether colors create pleasing relationships or inten- tional tension

    Color harmony vs. discord - whether colors create pleasing relationships or inten- tional tension

    work page

  6. [6]

    Overall color impact - how immediately the colors capture attention

    work page

  7. [7]

    Unusual or unexpected color choices - colors applied in unconventional ways

    work page

  8. [8]

    31 GPT-4o-mini Input Prompt For Image Evaluation (Part 3 of 3) “ EV ALUATION INSTRUCTIONS:

    Color temperature extremes - very warm or cool color schemes, or dramatic contrasts between warm and cool ” Figure 21: Input prompt passed into GPT-4o-mini to evaluate the AI-generated images in terms of both realism and color vividness and obtain two numerical scores (part 2 of 3). 31 GPT-4o-mini Input Prompt For Image Evaluation (Part 3 of 3) “ EV ALUAT...

    work page

Show all 43 references
  1. [10]

    Saturation levels - how pure and intense the colors appear

  2. [11]

    Color contrast - the degree of difference between adjacent colors

  3. [12]

    Relationship to natural color - how much colors deviate from their appearance in nature

  4. [18]

    First assess each dimension independently using their specific criteria

  5. [19]

    Then consider if the scores reflect an appropriate tradeoff relationship

  6. [20]

    Use decimal precision (e.g., 82.7 rather than just 83)

  7. [21]

    For images that seem to defy the usual tradeoff, verify if it truly represents excep- tional quality

  8. [22]

    Consider how this image would position relative to others on a Pareto frontier

  9. [23]

    Ensure you’re using the full scale range appropriately Output format: **Realism Score: [0.0-100.0]** **Color Boldness/Vividness Score: [0.0-100.0]** **Reasoning:** [Your detailed explanation, referencing specific visual elements and explain- ing how the image represents a spec...

  10. [24]

    Read a specific magazine’s requirements for their next cover image

  11. [25]

    View several AI-generated images

  12. [26]

    Use a type of feedback to guide the system toward better images

  13. [27]

    Select a final image that best matches the magazine’s requirements

  14. [28]

    Repeat steps 1-4 using four different types of feedback

  15. [29]

    Figure 23: MTurk User Study Instructions: Introduction, Study Overview, and Task Summary

    Complete a survey about your experience. Figure 23: MTurk User Study Instructions: Introduction, Study Overview, and Task Summary. 32 MTurk User Study Instructions (Part 2 of 6): Image Attributes Key Image Attributes The images in this study vary along two important dimensions...

  16. [30]

    • How to use it: Enter the numbers corresponding to each image in your preferred order

    Full Ranking • What it is: You will order all images from most preferred to least preferred, according to the magazine’s requirements. • How to use it: Enter the numbers corresponding to each image in your preferred order. • Example: If you have five images and prefer them in ...

  17. [31]

    • How to use it: Enter only the numbers for the top three images you have clear preferences about

    Partial Ranking • What it is: You will rank only the top three images based on your strongest prefer- ences according to the magazine’s requirements. • How to use it: Enter only the numbers for the top three images you have clear preferences about. • Example: If there is a tot...

  18. [32]

    • How to use it: Simply click on the image you prefer when presented with two options

    Pairwise Preferences • What it is: You will choose which image you prefer between two options at a time, according to the magazine’s requirements. • How to use it: Simply click on the image you prefer when presented with two options. Even if you do not like any of the images, ...

  19. [33]

    I reject any image below this threshold

    Soft & Hard Bounds • What it is: You’ll set minimum acceptable levels (hard bounds) and preferred levels (soft bounds) for image attributes. • How to use it: For each image quality attribute (realism and color vividness): – The left slider handle sets a hard bound (minimum acc...

  20. [34]

    Read the specific magazine requirements carefully. This will change for each type of feedback! • Example prompt interpretation: The example task description in an image (not shown here) will be looking for an image which provides a balance between realism and color vividness, ...

  21. [35]

    • The realism and color vividness scores are displayed below each image

    Review the initial set of images. • The realism and color vividness scores are displayed below each image

  22. [36]

    Provide feedback using the current mechanism

  23. [37]

    Review new images generated based on your feedback

  24. [38]

    pairwise, ranking, etc.) until you find a satisfactory image

    Using the same mechanism, provide feedback (i.e. pairwise, ranking, etc.) until you find a satisfactory image

  25. [39]

    Select Final Image

    Select your final image by clicking “Select Final Image” at the bottom of the page. • This will lead you to a page with all of the images you’ve seen displayed. Please enter in your level of satisfaction (0-100) with the final image, in terms of how appropriate you feel it is ...

  26. [40]

    {}” was replaced with the name of the specific feedback mechanism being evaluated (e.g., “Soft-Hard Bounds

    Move on to the next feedback mechanism. Figure 27: MTurk User Study Instructions: Step-by-step study process. MTurk User Study Instructions (Part 6 of 6): Tips & Final Notes Tips for Effective Image Selection • Keep the magazine’s requirements in focus throughout the selection...

  27. [41]

    {} takes a lot of mental effort to use

    Mental Effort: “{} takes a lot of mental effort to use.”

  28. [42]

    {} allows me to fully express my preferences

    Expressiveness: “{} allows me to fully express my preferences.”

  29. [43]

    I trust that my results with {} are optimal for its task

    Trust: “I trust that my results with {} are optimal for its task.” Figure 29: Survey questions presented to MTurk participants after their interaction with each feedback mechanism. The placeholder “{}” was substituted with the name of the mechanism in question. 37 MTurk User S...

  30. [151]

    URL https://proceedings.mlr.press/v216/benavoli23a.html

    PMLR, July 2023. URL https://proceedings.mlr.press/v216/benavoli23a.html. ISSN: 2640-3498. T. Broderick, A. Gelman, R. Meager, A. L. Smith, and T. Zheng. Toward a Taxonomy of Trust for Probabilistic Machine Learning, Dec. 2021. URL http://arxiv.org/abs/2112.03270. arXiv:2112.0...

    2023 arXiv

  31. [1959]

    Pages: xii, 153. S. Lundberg and S.-I. Lee. A Unified Approach to Interpreting Model Predictions, Nov. 2017. URL http://arxiv.org/abs/1705.07874. arXiv:1705.07874 [cs]. S. Luukkonen, H. W. van den Maagdenberg, M. T. M. Emmerich, and G. J. P. van Westen. Artificial intelligence...

    2017 arXiv

  32. [1967]

    Why Should I Trust You?

    ISSN 0001-4826. URL https://www.jstor.org/stable/243710. Publisher: American Accounting Association. M. T. Ribeiro, S. Singh, and C. Guestrin. "Why Should I Trust You?": Explaining the Predictions of Any Classifier, Aug. 2016. URL http://arxiv.org/abs/1602.04938. arXiv:1602.04...

    arXiv

  33. [2021]

    arXiv:2109.12750 [cs]

    URL http://arxiv.org/abs/2109.12750. arXiv:2109.12750 [cs]. G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions—I. Mathematical Programming, 14(1):265–294, Dec. 1978. ISSN 0025-5610, 1436-4646. doi: 10.1007/BF0...

    1978 arXiv

  34. [2023]

    doi: 10.1016/j.sbi.2023.102537

    ISSN 0959-440X. doi: 10.1016/j.sbi.2023.102537. URL https://www.sciencedirect. com/science/article/pii/S0959440X23000118. D. J. C. MacKay. Information-Based Objective Functions for Active Data Selection. Neural Computation, 4(4):590–604, July 1992. ISSN 0899-7667, 1530-888X. d...

    2023

  35. [9757]

    URL https://www.jair.org/index.php/jair/article/ view/10836

    doi: 10.1613/jair.3987. URL https://www.jair.org/index.php/jair/article/ view/10836. D. Sadigh, A. Dragan, S. Sastry, and S. Seshia. Active Preference-Based Learning of Reward Functions. In Robotics: Science and Systems XIII. Robotics: Science and Systems Foundation, July 2017...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.