Pith. sign in

REVIEW 1 major objections 7 minor 49 references

Utilizing Large Language Models to Synthesize Product Desirability Datasets

T0 review · 1 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that gpt-4o-mini can synthesize Product Desirability Toolkit word-and-review pairs whose sentiment matches the requested score, with correlations from 0.93 to 0.97.

desk verdict Useful, honest synthetic-PDT recipes with public data; the headline alignment numbers are self-consistency, not human fidelity. read the letter →

arxiv 2411.13485 v2 pith:2WXIJQ6S submitted 2024-11-20 cs.CL cs.AIcs.LG

John D. Hastings , Sherri Weitl-Harms , Joseph Doty , Zachary J. Myers , Warren Thompson This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords syntheticdatagenerationlargelanguagemodelsProductDesirabilityToolkitsentimentanalysisgpt-4o-miniuser-centereddesigntextdiversityaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a small, inexpensive language model can produce synthetic Product Desirability Toolkit (PDT) datasets—the word-plus-explanation pairs used to gauge whether users find a product desirable—at a scale real user testing rarely reaches. Using gpt-4o-mini and three prompt templates, it generated 1000 hypothetical software reviews per method and found sentiment scores tracking the intended scores with Pearson correlations from 0.93 to 0.97. Costs came to roughly $0.06 to $0.11 per thousand rows, which would make million-row datasets feasible for about $60 to $110. The credibility of this result rests on whether the model's own sentiment scores are a good proxy for human judgment, since the same model did most of the scoring; the authors flag direct comparison with human PDT datasets as future work.

What carries the argument

The central object is the Product Desirability Toolkit (PDT), a card-based method where users describe an experience by choosing words from a fixed 118-word reaction-card list and optionally explaining each choice. The machinery is a generate-and-score loop built on gpt-4o-mini: three prompt templates (Word+Review, Review+Word, Supply-Word) create 1000 word/review pairs each, and a 'Complete' scoring prompt assigns each pair a sentiment score between 0.00 and 1.00. Alignment is then measured by the Pearson correlation and absolute differences between those evaluated scores and the target scores, while diversity is measured by compression ratio, part-of-speech compression, homogenization, and n-gram diversity.

What would settle it

Have human raters score the same 3000 generated word/review pairs on the 0.00-1.00 sentiment scale used by the 'Complete' prompt and compute the correlation between human scores and the target scores; if that correlation is far below the reported 0.93-0.97, the alignment result is an artifact of self-scoring.

Watch

Extended reading notes

Core claim

The paper's central finding is that gpt-4o-mini can synthesize Product Desirability Toolkit word/review pairs whose evaluated sentiment closely tracks the intended target sentiment: Pearson correlations were 0.93 for Word+Review, 0.96 for Review+Word, and 0.97 for Supply-Word. The authors single out Review+Word and Supply-Word as more reliable for applications needing precise sentiment reflection. Supply-Word attained full coverage of the 118 PDT reaction cards and the most diverse text by compression-ratio, part-of-speech, homogenization, and n-gram measures, but it was the most verbose and most expensive method at about $0.11 per 1000 rows and showed the strongest positive bias. They attribute part of that positivity to the PDT word list's inherent 60/20/20 positive/negative/neutral makeup and note that differences between target and evaluated scores could arise in either the generation or the scoring phase; a gpt-4o check scoring of Supply-Word produced nearly identical patterns, suggesting generation, not scoring, drives most of the bias.

Load-bearing premise

The argument's load-bearing premise is that gpt-4o-mini's sentiment scores are a trustworthy stand-in for human judgment, since without that, the reported correlations only show the model agreeing with itself.

Editorial extensions

If this is right

  • Synthetic PDT datasets of 1000 rows can be produced for between $0.06 and $0.11 depending on method, so million-row datasets would cost roughly $60 to $110 in API charges.
  • Review+Word and Supply-Word, with Pearson correlations of 0.96 and 0.97, are the methods to prefer when the goal is precise reflection of a target sentiment.
  • Supply-Word gives the best coverage of the 118 PDT reaction cards and the most diverse vocabulary, at the price of higher cost and more positive bias.
  • Word+Review is the cheapest and fastest (about 1.5 seconds per review) but has the weakest alignment and sometimes invents words outside the PDT list; client-side validation can catch that.
  • All three methods carry a positive-sentiment bias, so synthetic data of this kind is recommended for internal research, not for real-world product, marketing, or public-relations decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the self-scoring caveat is resolved by human evaluation, the same generate-and-score loop would make it practical to benchmark LLM-based sentiment quantifiers against million-row PDT datasets, something the field currently lacks.
  • Because the prompts only name a product, the methods transfer to any product category or to other adjective-card toolkits, not just software.
  • The 79.3% of Supply-Word reviews starting with 'I recently' shows that standard diversity scores can look healthy while surface phrasing is repetitive; adding a naturalness or template-detection metric could change which method looks best.
  • A direct test of the bias question would be to weight target scores more heavily in the mid-range and see whether evaluated scores spread out, a possibility the authors mention but do not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper proposes three LLM-based methods for synthesizing Product Desirability Toolkit (PDT) datasets with gpt-4o-mini: Word+Review, Review+Word, and Supply-Word. Each method generates 1000 hypothetical software product reviews paired with PDT words and a sentiment score. The datasets are assessed in terms of sentiment alignment between target and evaluated scores, textual diversity (via compression ratio, part-of-speech compression, ROUGE-L homogenization, and n-gram diversity), and generation cost. The authors report high Pearson correlations (0.93–0.97) between target and evaluated scores, claim that Review+Word and Supply-Word show the best alignment, and conclude that LLM-generated synthetic PDT data is a scalable, cost-effective alternative when real data are scarce. The datasets are publicly released on Zenodo.

Significance. If the reported alignment reflects human-meaningful sentiment, the paper would offer a practical, low-cost pipeline for generating PDT-style data, addressing the acknowledged scarcity of large PDT datasets. The work is transparent in its methodology, publishes the generated datasets, provides detailed cost and token accounting, and uses standard diversity metrics. The main limitation is that the evaluation is not independent: the same model family that generates the reviews also scores them, so the reported correlations largely measure self-consistency rather than external fidelity. The paper's future-work section explicitly acknowledges that direct comparison with human PDT datasets remains to be done, and until such validation is provided, the central claim of high sentiment alignment is not fully supported.

major comments (1)
  1. [Section III-B.1 and Table IV] The Pearson correlations that support the central claim (0.93, 0.96, 0.97 in Table IV) are computed between the target score and a score produced by gpt-4o-mini using the 'Complete' prompt. For Supply-Word, the target score itself is also generated by gpt-4o-mini (Section III-A), so the correlation is essentially a self-consistency check of the same model family. For Word+Review and Review+Word, the target is externally supplied, but the evaluator is still the same model that wrote the review and selected the word; high correlations can therefore reflect the model's ability to follow its own scoring rubric rather than agreement with human sentiment. The paper's future-work section (Section V) concedes that direct comparison with human PDT datasets remains to be done. In light of this, the abstract's claim of 'high sentiment alignment' and the statement in Section IV-B that Review+Word and Supply-Word are 'more reliable for applications requiring precise sentiment reflection' are not supported by the current evidence. I recommend either tempering these claims or adding an external validation using human annotations or existing human PDT datasets.
minor comments (7)
  1. [Section I, RQ1] The phrase 'to to produce' contains a duplicated 'to'; it should read 'to produce'.
  2. [Section II] 'challenges inherit in data collection' should be 'challenges inherent in data collection'.
  3. [Section III-A] In the discussion of Word+Review and Review+Word, 'gpt-4o-min' is a typo for 'gpt-4o-mini', and 'resent' should be 're-sent'.
  4. [Table II] The 'Complete' prompt reads 'where is 0.00 is a completely negative sentiment'; the 'is' after 'where' should be removed.
  5. [Section IV-C and Table V] The finding that 79.3% of Supply-Word reviews start with 'I recently' while the diversity metrics (CR, CR-POS, HS, NDS) indicate the greatest diversity is counterintuitive. The paper acknowledges dataset-size differences as a caveat, but it would help to explain why such high repetition does not penalize the diversity scores.
  6. [Section IV-B, Table IV] The tStat column is reported without defining the test, the null hypothesis, or the expected range; the text says 'out of range tStat values warrant further exploration' but does not clarify what makes them out of range. Please add a definition and interpretation.
  7. [Section V, cost paragraph] The paragraph about OpenAI's lack of persistent session connections appears in Future Work but is really an infrastructure/cost detail; relocating it to Section IV-D would improve readability.

Circularity Check

2 steps flagged · score 6.0 of 10

For Supply-Word, the 'target' score is generated by the same gpt-4o-mini model that later evaluates the review, and the validity of model scoring rests on a self-citation to [37]; the reported alignment is substantially a self-consistency result, not an independent sentiment validation.

  1. self citation load bearing [Section III-B.1 (Assessing Alignment), scoring prompt selection]
    "given that gpt-4o and Claude were known to handle this approach well [37], and that this research does not aim to establish that gpt-4o-mini can properly score text using this specific approach – ultimately, all three methods were scored by asking gpt-4o-mini to analyze the review collectively based on the word and review."

    The only cited basis for treating LLM sentiment scores as accurate measures of PDT sentiment is [37], a prior paper by overlapping authors (Weitl-Harms, Hastings). The present study explicitly declines to validate gpt-4o-mini scoring ('does not aim to establish'). The headline Pearson correlations in Table IV (0.93, 0.96, 0.97) are computed between targets and gpt-4o-mini's own 'Complete'-prompt scores. Without an independent human anchor, the claim that these correlations show accurate sentiment alignment, and that Review+Word/Supply-Word are 'more reliable for applications requiring precise sentiment reflection', rests on the unverified prior self-citation rather than on evidence in this paper.

  2. self definitional [Section III-A.3 (Supply-Word) and Section III-B.1; Table IV]
    "The word was supplied to gpt-4o-mini, which was asked to score the sentiment expressed by the word, and produce a hypothetical software product review in line with that target score."

    For Supply-Word, the 'target' column of Table IV is not an externally supplied score; it is the output of gpt-4o-mini's implicit sentiment for the word. The review is then generated by the same model to match that target, and the 'evaluated' score is again produced by gpt-4o-mini from the word and review. The reported Pearson 0.97 is therefore corr(score_m(word), score_m(word, review_m(word, score_m(word)))). Both variables are outputs of the same model with the same sentiment prior, so the correlation measures self-consistency, not alignment with an independent definition of sentiment. Labeling the first column 'Expected' in Table IV is thus a misnomer for this method.

full rationale

Partial circularity. The paper's reported alignment scores are computed inside the same model that produced the data. The only cited warrant for using LLM scores as the sentiment criterion is [37], by overlapping authors, and the paper explicitly does not validate gpt-4o-mini scoring in this work. For Supply-Word, the 'target' is also a gpt-4o-mini output, making that method's 0.97 correlation a self-consistency statistic. Word+Review and Review+Word have externally randomized targets, so their 0.93/0.96 correlations are not forced by construction; they show the model can write text it scores at the requested level. But the conclusion that these methods are 'more reliable for applications requiring precise sentiment reflection' requires human validation, which the paper's future-work section concedes has not been done: it plans to generate synthetic data for products with human PDT datasets 'for which direct comparisons can be made'. Score 6 reflects partial circularity: one method's headline result is self-referential, and the evaluation model's validity is imported from an overlapping-author citation rather than demonstrated here.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim does not introduce fitted numerical parameters or new theoretical entities. It rests on the unvalidated assumption that LLM self-scoring approximates human sentiment, which is captured in the axioms above.

assumptions (3)
  • domain assumption gpt-4o-mini's 'Complete' prompt scores are treated as ground truth for the sentiment of generated reviews.
    Section III-B.1: all methods are scored using the 'Complete' prompt with gpt-4o-mini, and Supply-Word is additionally scored with gpt-4o. No human labels are used.
  • domain assumption The diversity metrics from Shaib et al. [43] are assumed to measure the human-representativeness of synthetic PDT text.
    Section III-B.2 and Table V use CR, CR-POS, HS, and NDS to rank methods; the paper does not connect these metrics to any human PDT responses.
  • domain assumption The PDT word list's 60% positive, 20% negative, 20% neutral composition is an acceptable prior for synthetic sentiment distributions.
    Section IV-A.2 invokes the word list skew to explain the positive bias in evaluated scores, but does not test whether that skew should govern synthetic data for downstream tools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Utilizing Large Language Models to Synthesize Product Desirability Datasets." pith.science (2026). https://pith.science/paper/2WXIJQ6S

@misc{pith2026241113485,
  author       = {Pith},
  title        = {Pith review of: Utilizing Large Language Models to Synthesize Product Desirability Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WXIJQ6S}},
  note         = {Machine review of arXiv:2411.13485}
}
read the original abstract

This research explores the application of large language models (LLMs) to generate synthetic datasets for Product Desirability Toolkit (PDT) testing, a key component in evaluating user sentiment and product experience. Utilizing gpt-4o-mini, a cost-effective alternative to larger commercial LLMs, three methods, Word+Review, Review+Word, and Supply-Word, were each used to synthesize 1000 product reviews. The generated datasets were assessed for sentiment alignment, textual diversity, and data generation cost. Results demonstrated high sentiment alignment across all methods, with Pearson correlations ranging from 0.93 to 0.97. Supply-Word exhibited the highest diversity and coverage of PDT terms, although with increased generation costs. Despite minor biases toward positive sentiments, in situations with limited test data, LLM-generated synthetic data offers significant advantages, including scalability, cost savings, and flexibility in dataset production.

Figures

Figures reproduced from arXiv: 2411.13485 by the authors.

Figure 1
Figure 1. Histogram of Evaluated PDT Scores. The tendency toward the ends of the range is even more pronounced for Supply-Word, whether scored by gpt-4o-mini or gpt-4o. Part of this was caused by the Supply-Word ap￾proach, which supplied a word to gpt-4o-mini for scoring. None of these words scored as a 0.0 (unlike Word+Review and Review+Word), resulting in higher target scores and, consequently, higher evaluated scores. Give… view at source ↗
Figure 2
Figure 2. Distribution of Absolute Differences between Target & Evaluated Scores per Method by Percentage. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 34 canonical work pages

  1. [1]

    Synthetic data generator for classification rules learning,

    R. Liu, B. Fang, Y . Y . Tang, and P. P. Chan, “Synthetic data generator for classification rules learning,” in 2016 7th International Conference on Cloud Computing and Big Data (CCBD) , 2016, pp. 357–361. DOI: 10.1109/CCBD.2016.076

    work page doi:10.1109/ccbd.2016.076 2016

  2. [2]

    Exploring large language models for low-resource IT information extraction,

    B. Bhavya, P. T. Isaza, Y . Deng, et al. , “Exploring large language models for low-resource IT information extraction,” in 2023 IEEE International Conference on Data Mining Workshops (ICDMW), 2023, pp. 1203–1212. DOI: 10.1109/ICDMW60847.2023.00157

    arXiv 2023

  3. [3]

    Synthetic datasets generator for testing information visualization and machine learning techniques and tools,

    S. D. P. Mendonc ¸a, Y . P. D. S. Brito, C. G. R. D. Santos, R. D. A. D. Lima, T. D. O. D. Ara ´ujo, and B. S. Meiguins, “Synthetic datasets generator for testing information visualization and machine learning techniques and tools,” IEEE Access, vol. 8, pp. 82 917–82 928, 2020. DOI: 10.1109/ACCESS.2020.2991949

    work page arXiv 2020

  4. [4]

    Criteria for a comparative study of visualization techniques in data mining,

    R. Redpath and B. Srinivasan, “Criteria for a comparative study of visualization techniques in data mining,” Intelligent Systems Design and Applications , pp. 609–620, 2003. DOI: 10 . 1007 / 978 - 3 - 540 - 44999-7 58

    work page 2003

  5. [5]

    Between level up and game over: A systematic literature review of gamification in education,

    A. Manzano-Le ´on, P. Camacho-Lazarraga, M. A. Guerrero, et al. , “Between level up and game over: A systematic literature review of gamification in education,” Sustainability, vol. 13, no. 4, p. 2247, 2021. DOI: 10.3390/su13042247

    doi:10.3390/su13042247 2021

  6. [6]

    Generation and evaluation of synthetic patient data,

    G. Andre, P. R. Braden, S. Jennifer, S. L. Coyle, and A. P. Sales, “Generation and evaluation of synthetic patient data,” BMC medical research methodology, vol. 1, no. 20, pp. 1–40, 2020. DOI: 10.1186/ s12874-020-00977-1

    work page 2020

  7. [7]

    A review of feature selection methods on synthetic data,

    V . Bol ´on-Canedo, N. S ´anchez-Maro˜no, and A. Alonso-Betanzos, “A review of feature selection methods on synthetic data,” Knowledge and Information Systems, vol. 34, no. 3, 2013. DOI: 10.1007/s10115-012- 0487-8

    work page doi:10.1007/s10115-012- 2013

  8. [8]

    Data generators: A short survey of techniques and use cases with focus on testing,

    I. V . S. Popi ´c B. Pavkovi ´c and N. Tesli ´c, “Data generators: A short survey of techniques and use cases with focus on testing,” in Proc. IEEE 9th Int. Conf. Consum. Electron , 2019, pp. 189–194. DOI: 10. 1109/ICCE-Berlin47944.2019.8966202

    work page arXiv 2019

Show all 49 references
  1. [9]

    Development of a synthetic data set generator for building and testing information discovery systems,

    P. J. Lin, B. Samadi, A. Cipolone, et al., “Development of a synthetic data set generator for building and testing information discovery systems,” in Proc. 3rd Int. Conf. Inf. Technol. New Gener. (ITNG) , 2006, pp. 707–712. DOI: 10.1109/ITNG.2006.51

    2006 doi

  2. [10]

    Analysis of Facebook interaction as basis for synthetic expanded social graph generation,

    L. Humski, D. Pintar, and M. Vrani ´c, “Analysis of Facebook interaction as basis for synthetic expanded social graph generation,” IEEE Access, vol. 7, pp. 6622–6636, 2019. DOI: 10.1109/ACCESS.2018.2886468

    2019

  3. [11]

    Generating datasets with pretrained lan- guage models,

    T. Schick and H. Sch ¨utze, “Generating datasets with pretrained lan- guage models,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds., Online and Punta Cana, Domini- can Republi...

    2021 doi

  4. [12]

    Sentiment analysis in online product reviews: Mining customer opinions for sentiment classification,

    L. Bharadwaj, “Sentiment analysis in online product reviews: Mining customer opinions for sentiment classification,” Int J Multidiscip Res , vol. 5, no. 5, 2023. DOI: 10.36948/ijfmr.2023.v05i05.6090

    2023 doi

  5. [13]

    Do not have enough data? Deep learning to the rescue!

    A. Anaby-Tavor, B. Carmeli, E. Goldbraich, et al. , “Do not have enough data? Deep learning to the rescue!” In Proceedings of the AAAI conference on artificial intelligence , vol. 34, 2020, pp. 7383–7390. DOI: 10.1609/aaai.v34i05.6233

    2020 doi

  6. [14]

    Synthesize step-by-step: Tools, templates and LLMs as data generators for reasoning-based chart VQA,

    Z. Li, B. Jasani, P. Tang, and S. Ghadar, “Synthesize step-by-step: Tools, templates and LLMs as data generators for reasoning-based chart VQA,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 13 613–13 623. DOI: 10.1109/ CVPR52733.2024.01292

    2024

  7. [15]

    ISO, Ergonomics of human-system interaction – part 11: Usability: Definitions and concepts , ISO 9241-11, Standard, International Orga- nization for Standardization (ISO), 2018

    2018

  8. [16]

    Clustering to categorize desir- ability in software: Exploring cluster analysis of product reaction cards in a stereoscopic retail application,

    D. Gonz ´alez-Z´u˜niga and J. Carrabina, “Clustering to categorize desir- ability in software: Exploring cluster analysis of product reaction cards in a stereoscopic retail application,” in 2016 Digital Media Industry and Academic Forum (DMIAF) , 2016, pp. 193–197. DOI: 10.110...

    2016

  9. [17]

    A supporting tool for enhancing user’s mental model elicitation and decision-making in user experi- ence research,

    M. Mart ´ın and J. A. Mac ´ıas, “A supporting tool for enhancing user’s mental model elicitation and decision-making in user experi- ence research,” International Journal of Human–Computer Interaction, vol. 39, no. 1, pp. 183–202, 2023. DOI: 10 . 1080 / 10447318 . 2022 . 2041885

    2023

  10. [18]

    Mapping customer needs to design parameters in the front end of product design by applying deep learning,

    Y . Wang, D. Y . Mo, and M. M. Tseng, “Mapping customer needs to design parameters in the front end of product design by applying deep learning,” CIRP Annals, vol. 67, no. 1, pp. 145–148, 2018, ISSN : 0007-8506. DOI: https://doi.org/10.1016/j.cirp.2018.04.018

    2018 doi

  11. [19]

    Analysis of sentiment expressions for user-centered design,

    Y . Han and M. Moghaddam, “Analysis of sentiment expressions for user-centered design,” Expert Systems with Applications , vol. 171, p. 114 604, 2021, ISSN : 0957-4174. DOI: https : / / doi . org / 10 . 1016 / j.eswa.2021.114604

    2021

  12. [20]

    Deep learning for sentiment analysis: A survey,

    L. Zhang, S.Wang, and B.Liu, “Deep learning for sentiment analysis: A survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, no. 4, 2018. DOI: 10.1002/widm.1253

    2018 doi

  13. [22]

    Benedek and T

    J. Benedek and T. Miner, Product reaction cards , 2002. [Online]. Available: https://web.archive.org/web/20030709123349/http://www. microsoft.com/usability/UEPostings/ProductReactionCards.doc

    2002

  14. [23]

    More than a feeling: Understanding the desirability factor in user experience,

    C. M. Barnum and L. A. Palmer, “More than a feeling: Understanding the desirability factor in user experience,” in CHI ’10 Extended Abstracts on Human Factors in Computing Systems , Association for Computing Machinery, 2010. DOI: 10.1145/1753846.1754217

    2010

  15. [24]

    Barnum, Usability Testing Essentials

    C. Barnum, Usability Testing Essentials . Elsevier, 2020, ISBN : 9780128169438

    2020

  16. [25]

    End-user experiences of visual and textual programming environments for Arduino,

    T. Booth and S. Stumpf, “End-user experiences of visual and textual programming environments for Arduino,” in IS-EUD, 2013. DOI: 10. 1007/978-3-642-38706-7 4

    2013

  17. [26]

    CARMA: Assessing usability through a non-biased online survey technique,

    J. D. Hastings, A. Mirasano, A. Latchininsky, and S. P. Schell, “CARMA: Assessing usability through a non-biased online survey technique,” in 43rd Hawaii International Conference on System Sci- ences, 2010, pp. 1–10. DOI: 10.1109/HICSS.2010.93

    2010 doi

  18. [27]

    Mobile interface studies about style description and influential factors,

    Y . Li and X.-y. Wang, “Mobile interface studies about style description and influential factors,” in Int. Conf. on Management Science & Engineering, 2014. DOI: 10.1109/ICMSE.2014.6930281

    2014

  19. [28]

    Tullis and B

    T. Tullis and B. Albert, Measuring the User Experience . Morgan Kaufmann, 2013

    2013

  20. [29]

    Supporting user-perceived usability bench- marking through a developed quantitative metric,

    R. Veral and J. A. Mac ´ıas, “Supporting user-perceived usability bench- marking through a developed quantitative metric,”Int. J. Hum. Comput. Stud., vol. 122, 2019. DOI: 10.1016/j.ijhcs.2018.09.012

    2019 doi

  21. [30]

    Assessing user experiences with ZORQ: A gamification framework for computer science education,

    S. Weitl-Harms, A. Spanier, J. Hastings, and M. Rokusek, “Assessing user experiences with ZORQ: A gamification framework for computer science education,” in Hawaii International Conference on System Sciences, 2023. [Online]. Available: https : / / aisel . aisnet . org / hicss ...

    2023

  22. [31]

    Lewis and J

    J. Lewis and J. Sauro, 10 things to know about the microsoft desir- ability toolkit , 2020. [Online]. Available: https : / / measuringu . com / microsoft-desirability (visited on 04/10/2022)

    2020

  23. [32]

    Evaluation of information visualization techniques: Analysing user experience with reaction cards,

    T. Mer ˇcun, “Evaluation of information visualization techniques: Analysing user experience with reaction cards,” in Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, ser. BELIV ’14, Paris, France: Association for Computin...

    2014

  24. [33]

    Chapter 8. capturing sensory experiences through semi-structured elicitation questions,

    U. Gretzel and D. R. Fesenmaier, “Chapter 8. capturing sensory experiences through semi-structured elicitation questions,” in Con- sumer and Managerial Perspectives , M. Morgan, P. Lugosi, and J. B. Ritchie, Eds. Bristol, Blue Ridge Summit: Channel View Publications, 2010, pp....

    2010

  25. [34]

    Conceptualizing experience: A tourist based approach,

    S. V olo, “Conceptualizing experience: A tourist based approach,” Journal of Hospitality Marketing & Management , vol. 18, no. (2/3), pp. 111–126, 2009. DOI: 10.1080/19368620802590134

    2009 doi

  26. [35]

    Developing an instrument to capture multifaceted visitor experiences: The dove adjective checklist,

    R. B. Jan Packer and N. Bond, “Developing an instrument to capture multifaceted visitor experiences: The dove adjective checklist,” Visitor Studies, vol. 21, no. 2, pp. 211–231, 2018. DOI: 10.1080/10645578. 2018.1553925

    2018

  27. [36]

    Cracking the code of mass customization,

    F. Salvador, P. M. De Holan, and F. T. Piller, “Cracking the code of mass customization,” MIT sloan management review , 2009. [Online]. Available: https://sloanreview.mit.edu/article/cracking- the- code- of- mass-customization (visited on 10/26/2024)

    2009

  28. [37]

    Using LLMs to establish implicit user sentiment of software desirability,

    S. Weitl-Harms, J. D. Hastings, and J. Lum, “Using LLMs to establish implicit user sentiment of software desirability,” in Proceedings of the 2024 IEEE International Conference on Machine Learning and Applications (ICMLA-24) , in press , Dec. 2024. arXiv: 2408 . 01527 [cs.CL]

    2024

  29. [38]

    Deriving future robo-taxi ux keywords using PRC (product- reaction cards),

    D. Lim, “Deriving future robo-taxi ux keywords using PRC (product- reaction cards),” Design Convergence Study, vol. 19, no. 6, 2020

    2020

  30. [39]

    Are reference pop-up widgets welcome or annoying? a usability study,

    B. Imler, K. Garcia, and N. Clements, “Are reference pop-up widgets welcome or annoying? a usability study,” Reference Services Review , vol. 44, no. 3, 2016. DOI: 10.1108/RSR-11-2015-0049

    2016 doi

  31. [40]

    ZORQ: A gamification framework for computer science education,

    J. D. Hastings, S. Weitl-Harms, A. Spanier, M. Rokusek, and R. Henszey, “ZORQ: A gamification framework for computer science education,” in 2022 IEEE Frontiers in Education Proceedings , 2022, pp. 1–9. DOI: 10.1109/FIE56618.2022.9962487

    2022

  32. [41]

    [Online]

    OpenAI, Gpt-4o mini: Advancing cost-efficient intelligence , 2024. [Online]. Available: https://openai.com/index/gpt-4o-mini-advancing- cost-efficient-intelligence (visited on 10/23/2024)

    2024

  33. [42]

    [Online]

    OpenAI, Openai api platform , 2024. [Online]. Available: https : / / openai.com/api/ (visited on 10/03/2024)

    2024

  34. [43]

    Shaib, J

    C. Shaib, J. Barrow, J. Sun, A. F. Siu, B. C. Wallace, and A. Nenkova, Standardizing the measurement of text diversity: A tool and a comparative analysis of scores, 2024. arXiv: 2403.00553 [cs.CL]

    2024

  35. [44]

    Hastings, S

    J. Hastings, S. Weitl-Harms, J. Doty, Z. Myers, and W. Thompson, Synthetic product desirability datasets for sentiment analysis testing (1.0.0), 2024. DOI: 10.5281/zenodo.14188456

    2024 doi

  36. [45]

    [Online]

    OpenAI, OpenAI pricing , 2024. [Online]. Available: https://openai. com/api/pricing/ (visited on 10/23/2024)

    2024

  37. [46]

    Energy and policy con- siderations for deep learning in NLP,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy con- siderations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3645–...

    2019 doi

  38. [47]

    Green AI,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM , vol. 63, no. 12, pp. 54–63, Nov. 2020. DOI: 10.1145/3381831

    2020 doi

  39. [48]

    Aligning artificial intelligence with climate change mitigation,

    L. H. Kaack, P. L. Donti, E. Strubell, G. Kamiya, F. Creutzig, and D. Rolnick, “Aligning artificial intelligence with climate change mitigation,” Nature Climate Change, vol. 12, no. 6, pp. 518–527, 2022. DOI: 10.1038/s41558-022-01377-7

    2022 doi

  40. [49]

    Generative AI’s environmental costs are soaring — and mostly secret,

    K. Crawford, “Generative AI’s environmental costs are soaring — and mostly secret,” Nature, vol. 626, no. 693, 2024. DOI: 10.1038/d41586- 024-00478-x

    2024 doi

  41. [50]

    CARMA: A case-based rangeland management adviser,

    J. Hastings, K. Branting, and J. Lockwood, “CARMA: A case-based rangeland management adviser,” AI Magazine, vol. 23, no. 2, pp. 49– 62, 2002. DOI: 10.1609/aimag.v23i2.1640

    2002 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.