Pith. sign in

REVIEW 5 major objections 7 minor 53 references

Suspicious AI-assisted survey answers leave most numbers stable but reshape how qualitative findings are framed and evidenced.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:52 UTC pith:BV4VVKPR

load-bearing objection Useful differential finding (quant mostly stable, qual framing more movable) from four real SE surveys, undercut a bit by manual-label ground truth and a clear arithmetic slip in the flagship Global South example. the 5 major comments →

arxiv 2607.23805 v1 pith:BV4VVKPR submitted 2026-07-26 cs.SE

The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys

Ronnie de Souza Santos , Italo Santos , Maria Teresa Baldassarre , Cleyton Magalhaes , Mairieli Wessel This is my paper
classification cs.SE
keywords LLMssurveythreats to validitysoftware engineeringAI-generated responsesdata qualitythematic analysisonline recruitment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether LLM-assisted or otherwise fraudulent answers quietly undermine software engineering surveys. Reanalyzing four real online surveys, the authors remove responses flagged by careful manual review and compare what remains with the original data. Closed-ended and demographic patterns mostly hold, with only moderate shifts on some group and perception measures. Open-ended themes keep their broad shape, but the stories that drive interpretation change: polished, extreme, or formulaic narratives drop out, so context, code emphasis, and supporting quotes look different. The practical claim is that AI-assisted participation already shows up in SE surveys and hits qualitative work harder than counts, so validation must mix manual inspection with other checks rather than trust bots or detectors alone.

Core claim

Across four software engineering survey datasets, filtering manually identified suspicious responses left broader quantitative distributions largely intact, with moderate shifts on some demographic and analytical variables, while qualitative findings changed more through altered contextual framing, category prominence, and the nature of the evidence used to characterize participant experiences. AI-assisted participation therefore threatens validity differently by analysis type, and multiple complementary validation procedures—especially manual review of open-ended text—are needed.

What carries the argument

Paired original-versus-manually-cleaned comparison: two researchers consensus-label suspicious open-ended answers using cross-response stylistic and narrative indicators, then re-run the same descriptive statistics and thematic analysis on both versions (with an automated AI text detector used only as a secondary check).

Load-bearing premise

The dual-researcher manual labels of “suspicious” answers are treated as good enough ground truth for what counts as fraudulent or AI-assisted participation when measuring impact.

What would settle it

On similar SE surveys, if independent validation (e.g., known-human control responses, stronger provenance, or higher-agreement multi-rater labeling) shows that removing differently labeled sets does not change qualitative framing or evidence while quantitative patterns stay stable—or shows large quantitative breaks when true AI answers are removed—the reported stability-versus-fragility contrast would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SE survey papers that lean on open-ended themes should treat AI-assisted participation as a real validity threat, not only a theoretical one.
  • Broader closed-ended conclusions may survive moderate contamination, but claims about underrepresented groups or specific perception items can still shift by several percentage points.
  • Automated AI detectors alone are insufficient: they miss short answers, disagree with manual review, and produce false positives.
  • Authors should document how they inspect narratives, handle suspicious cases, and whether filtering changed interpretation.
  • Future work should test how different fractions of AI-assisted answers change quantitative distributions and qualitative readings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Qualitative SE studies that quote vivid hardship, integrity, or optimization stories may be most exposed, because those are exactly the polished patterns the cleaned data de-emphasized.
  • Recruitment on open marketplaces may need narrative cross-checks as standard as attention items once LLMs are cheap for respondents.
  • A useful follow-on experiment would inject known LLM answers at controlled rates into real SE instruments and measure when thematic framing flips versus when percentages move.
  • Reporting both full and cleaned qualitative codebooks could become a practical transparency norm when open text drives the claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The manuscript reports a meta-scientific secondary analysis of four software engineering survey datasets (Global South education, N=81; student AI cheating, N=145; safe spaces, N=133; professional LLM usage, N=128), all recruited via Prolific. Two researchers manually flagged suspicious responses (19/29/41/9 per dataset) using multi-indicator consensus informed by the authors' prior exploratory study [8]; the Scribbr AI detector was run for comparison. The authors then re-computed descriptive statistics and re-ran thematic analyses on cleaned datasets. They find quantitative distributions generally stable (shifts of ~3–6 percentage points on some demographic/analytical variables), while qualitative interpretation is more strongly affected through changes in code prominence, contextual framing, and the trustworthiness of illustrative quotes. They conclude automated detectors are unreliable alone (many short responses unanalyzable; false positives both directions) and recommend combined validation procedures, especially manual inspection of open-ended text.

Significance. If the results hold, this is a useful and timely methodological contribution: it provides concrete evidence that AI-assisted participation is already present in Prolific-recruited SE surveys despite platform controls, and it documents an asymmetry — robustness of aggregate proportions vs sensitivity of qualitative framing — that survey researchers should internalize. Strengths worth crediting: four distinct survey contexts; dual independent coding with consensus for both labeling and thematic analysis; transparent reporting of detector disagreements in both directions (false positives and false negatives); a full re-run of the thematic analysis on cleaned data rather than a token sensitivity check; hedged claims; and a promised replication package. The quantitative tables are internally consistent to the unit level, which raises confidence in the underlying data even where the qualitative reporting falters.

major comments (5)
  1. [§4.3.1 (Global South qualitative)] Arithmetic in the flagship qualitative example does not reconcile. The nine original categories sum to 74 (25+16+6+6+6+5+5+3+2), not the stated 81 core ideas. Resource and Infrastructure Scarcity is enumerated as 16 codes but then 'decreased from 20 to 10' — the most-emphasized category collapse is materially overstated if 16 is correct. Cleaned counts also show increases (Transportation 6→7, Academic Continuation 3→6), which is impossible under pure removal; the other three datasets satisfy exactly one code per response with purely subtractive deltas (145→116, 133→92). Two categories vanish without comment; Financial Constraints is unchanged at 25 despite 19 removals. Please re-derive from the raw coding data, state the coding convention, and state whether cleaned coding was subtractive or de novo.
  2. [§4.1 (manual vs automated detection)] The manual-vs-detector counts do not form a clean partition of N in any dataset. Global South: legitimate components (1 flagged 100% + 15 at 0% + 47 unanalyzable) sum to 63 vs 62 legitimate. Student Cheating: 13+1+2+7 flagged plus 7 at 0% gives 30 vs 29 suspicious; 54 at 0% + 68 unanalyzable = 122 vs 116 legitimate. Safe Spaces accounts for only 23 of 41 suspicious and 81 of 92 legitimate; LLM Usage accounts for 8 of 9 suspicious and 75 of 119 legitimate. Since contribution (3) concerns detector limitations, please provide a full cross-tabulation per dataset (manual label × detector outcome, including unanalyzable) so readers can verify agreement structure.
  3. [§3.3 (manual identification); §3.4 (construct validity)] Every impact estimate is defined relative to the manual 'suspicious' labels, yet no pre-consensus agreement statistic (kappa or raw agreement, number of disagreements) is reported, and the replication package should include per-response indicator rationales. Please also state the temporal order of labeling vs thematic coding of the original data and whether coders were blind to suspicion labels during re-coding of the cleaned data — otherwise the stability/fragility contrast could partially reflect the labeling rule or coder expectancy, including misclassification of polished non-native-English writing.
  4. [§3.3; relationship to reference [8]] The indicator set was 'informed by patterns reported in prior exploratory work' [8], which is the same author team's study of AI-generated responses in SE surveys. Please state explicitly whether [8] analyzed these same four datasets. If so, the detection is partly confirmatory on the data from which the heuristics were induced, and the prevalence and contrast findings should not be read as independent validation. Also align terminology: the title asserts 'Fraudulent AI-Generated Responses' while the body carefully uses 'suspicious or potentially AI-assisted'; without ground truth, the title overstates what is measured.
  5. [§4.2.5, §4.3, Discussion] The headline contrast — quantitative 'generally stable' vs qualitative 'more strongly influenced' — lacks an operational criterion. A 6.4pp drop in Black participants (§4.2.2) and a 5.7pp drop in safe-space awareness (§4.2.3) are arguably as consequential for the original studies' conclusions as a 23→11 code-count reduction; the asymmetry may partly reflect measurement granularity (proportions on fixed items vs open-ended code frequencies). Please state the criterion for 'stable' vs 'influenced' (e.g., would any stated conclusion of the original studies change?) or soften the contrast.
minor comments (7)
  1. [§3.1] Broken citation: 'attention validation questions embedded throughout the surveys [30, 32, ?]' — unresolved reference marker remains in the text.
  2. [§4.1] Detector probabilities are reported with inconsistent precision ('0.19%' vs '32%, 64%, 75%'); 0.19% is also listed among suspicious-set intermediates but later attributed to a legitimate response. Standardize units and attribution.
  3. [§4.3.1] Sentence fragment: 'the remaining dataset still preserved consistent evidence ... through more grounded, such as students reporting not enough computers' — missing noun after 'grounded'.
  4. [§4.1] The detector's word-threshold failures are a headline result worth quantifying in one place: roughly 60–85% of manually legitimate responses per dataset were unanalyzable. This is arguably the strongest practical evidence for contribution (3) and is currently scattered.
  5. [§3 / §4.3] No ethics/consent statement for secondary analysis of participant narratives, and no discussion of whether reproducing quotes from responses judged fraudulent raises provenance concerns. A brief statement would suffice.
  6. [§3.3] No agreement statistic is reported for the thematic coding either (only consensus is described). A pre-consensus agreement figure for at least one dataset would strengthen the qualitative re-analysis.
  7. [§7 (Data Availability)] Verify the figshare replication package is publicly accessible at publication, and restore the withheld citations to the original surveys in the camera-ready as promised.

Circularity Check

1 steps flagged

No derivation-by-construction circularity; only mild non-load-bearing self-citation of the authors’ prior indicator set for labeling “suspicious” responses.

specific steps
  1. self citation load bearing [§3.3 Manual identification of suspicious responses; also Background/Related via [8]]
    "The manual identification process was informed by patterns reported in prior exploratory work on AI-generated responses in software engineering surveys [8]. During the review process, the researchers evaluated multiple indicators jointly rather than relying on a single characteristic in isolation. These indicators included repetitive narrative structures across participants, highly similar stylistic composition and rhetorical organization, reused or formulaic expressions, superficial personalization, duplicated semantic structures..."

    The operational definition of the cleaned datasets rests on manual labels whose indicator set is taken from the same team’s prior paper [8] (same core author list). Impact of “suspicious” responses is then defined as whatever changes after removing those labels. This is mild self-citation dependence on the labeling rule, not a forced mathematical reduction: the paper does not claim the labels are uniquely determined by an external theorem, and the original-vs-cleaned quantitative/qualitative contrasts remain independently reported empirical measurements. Automated detector comparison and explicit FP/FN threats further keep the central claim from collapsing into the self-citation alone.

full rationale

This is an empirical secondary-analysis paper, not a first-principles derivation. The central claim—that filtering responses the authors manually labeled suspicious leaves broader quantitative distributions largely stable while more strongly shifting qualitative framing and evidence—is obtained by a before/after comparison on four datasets, not by fitting a parameter and renaming the fit as a prediction, nor by importing a uniqueness theorem that forces the result. The only circularity-adjacent element is that manual “suspicious” labels (which define the cleaned datasets) were informed by patterns from the authors’ own prior exploratory paper [8], and impact is then measured by removing those same labels. That is ordinary method dependence / operational ground-truth choice, acknowledged in Threats (§3.4) with false-positive/false-negative caveats, and cross-checked against an external automated detector. It does not make the original-vs-cleaned contrasts tautological: the contrasts still report independent empirical content (which variables and themes move, by how much). No fitted-input-called-prediction, no self-definitional equation collapse, no uniqueness import, no ansatz smuggled as theorem. Score 2 reflects one minor self-citation that is not load-bearing for the comparative findings.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The central contrast (quantitative stability vs qualitative sensitivity) rests on treating dual-coder manual “suspicious” labels as the cleaning criterion, on secondary use of four author-collected Prolific surveys, on inductive thematic analysis as the qual instrument, and on descriptive proportion comparisons without a formal causal model of fraud. No new physical entities; the load-bearing commitments are methodological domain assumptions and one ad hoc operationalization of fraud.

free parameters (3)
  • Manual suspicious-response decision threshold (multi-indicator consensus) = Binary consensus label per open-ended response (counts: 19/81, 29/145, 41/133, 9/128)
    No numeric cutoff; binary include/exclude per response is a human judgment rule that directly defines cleaned N and all downstream deltas.
  • Scribbr AI-detector probability cutoffs used for comparison = Emphasis on 100% flags plus assorted intermediate scores
    Intermediate probabilities (e.g., 32–96%) are reported narratively; “100% AI” is treated as strong automated flag without a pre-registered decision threshold for agreement metrics.
  • Subset of 24 demographic/closed/open questions selected for reanalysis = 24 questions across four surveys
    Authors chose which items represent “central constructs”; different selection could change which variables show 3–6 pp shifts.
axioms (7)
  • domain assumption Recurring cross-response stylistic similarity, formulaic polish, and implausible detail are valid indicators of fraudulent or AI-assisted participation rather than merely good writing or shared culture.
    Invoked in §3.3 manual identification; underpins the cleaned datasets.
  • domain assumption Secondary analysis of existing survey datasets can answer new methodological RQs about response authenticity without re-collecting data.
    Stated in §3 meta-science framing and citations to secondary-analysis methods.
  • domain assumption Descriptive frequency/proportion shifts and inductive thematic re-coding are sufficient to assess whether findings “remain stable” or are “influenced.”
    §3.3 quantitative and qualitative analysis design; no inferential model required by the paper’s claims.
  • domain assumption Commercial AI-text detectors provide probabilistic signals useful for comparison but not ground truth.
    §3.3 and Results §4.1; authors correctly treat Scribbr as non-definitive.
  • domain assumption Prolific-recruited samples with platform quality controls are an appropriate setting in which to study contemporary SE survey fraud risk.
    §3.1 study contexts; limits external validity as noted in §3.4.
  • standard math Standard qualitative thematic analysis steps (familiarization, coding, categories, themes) with dual-author consensus yield comparable theme structures across original and cleaned corpora.
    Method cites Clarke & Braun / SE qualitative guidance; used as accepted procedure rather than a new formal system.
  • ad hoc to paper Operational definition equating “manually suspicious” with the responses whose removal defines the effect of AI-assisted participation.
    Core design choice of the paper; not independently validated against known AI-generated holdout answers.

pith-pipeline@v1.2.0-grok45-kimik3 · 20862 in / 4012 out tokens · 83364 ms · 2026-07-30T11:52:02.351861+00:00 · methodology

0 comments
read the original abstract

Background: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys. Aims: This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings. Method: We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results: Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants' experiences and study contexts were interpreted and characterized. Conclusions: AI assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open ended responses.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 3 linked inside Pith

  1. [1]

    Proceedings of the ACM on Software Engineering , volume=

    Can gpt-4 replicate empirical software engineering research? , author=. Proceedings of the ACM on Software Engineering , volume=. 2024 , publisher=

    2024

  2. [2]

    Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , pages=

    The promise and challenges of using LLMs to accelerate the screening process of systematic reviews , author=. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , pages=

  3. [3]

    Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

    ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity , author=. Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

  4. [4]

    Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

    Can ChatGPT emulate humans in software engineering surveys? , author=. Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

  5. [5]

    ACM Transactions on Software Engineering and Methodology , volume=

    Large language models for software engineering: A systematic literature review , author=. ACM Transactions on Software Engineering and Methodology , volume=. 2024 , publisher=

    2024

  6. [6]

    Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization , pages=

    Simulating Human Opinions with Large Language Models: Opportunities and Challenges for Personalized Survey Data Modeling , author=. Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization , pages=

  7. [7]

    2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=

    Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering , author=. 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=. 2025 , organization=

    2025

  8. [8]

    Proceedings of the National Academy of Sciences , volume=

    Transforming literature screening: The emerging role of large language models in systematic reviews , author=. Proceedings of the National Academy of Sciences , volume=. 2025 , publisher=

    2025

  9. [9]

    Information , volume=

    Exploring the Use and Misuse of Large Language Models , author=. Information , volume=. 2025 , publisher=

    2025

  10. [10]

    2025 , school=

    LLM Cheat Prevention Via Adversarial Question Paraphrasing , author=. 2025 , school=

    2025

  11. [11]

    2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=

    On the difficulties of conducting and replicating systematic literature reviews studies using LLMs in software engineering , author=. 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=. 2025 , organization=

    2025

  12. [12]

    A Large-Scale Replication of Psychological Experiments with LLMs (August 25, 2024) , year=

    Can ai replace human subjects? a large-scale replication of psychological experiments with llms , author=. A Large-Scale Replication of Psychological Experiments with LLMs (August 25, 2024) , year=

    2024

  13. [13]

    2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=

    Towards evaluation guidelines for empirical studies involving llms , author=. 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) , pages=. 2025 , organization=

    2025

  14. [14]

    Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , pages=

    Breaking the silence: the threats of using llms in software engineering , author=. Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , pages=

    2024

  15. [15]

    2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) , pages=

    Large language models for software engineering: Survey and open problems , author=. 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) , pages=. 2023 , organization=

    2023

  16. [16]

    Proceedings of the 55th ACM Technical Symposium on Computer Science Education V

    Software engineering education must adapt and evolve for an llm environment , author=. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 , pages=

  17. [17]

    arXiv preprint arXiv:2404.01268 , year=

    Mapping the increasing use of LLMs in scientific papers , author=. arXiv preprint arXiv:2404.01268 , year=

    Pith/arXiv arXiv

  18. [18]

    International Research Journal of Modernization in Engineering Technology and Science , volume=

    Contribution and performance of ChatGPT and other Large Language Models (LLM) for scientific and research advancements: a double-edged sword , author=. International Research Journal of Modernization in Engineering Technology and Science , volume=

  19. [19]

    IEEE Access , volume=

    Survey research in software engineering: Problems and mitigation strategies , author=. IEEE Access , volume=. 2018 , publisher=

    2018

  20. [20]

    Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement , pages=

    Survey guidelines in software engineering: An annotated review , author=. Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement , pages=

  21. [21]

    ACM SIGSOFT Software Engineering Notes , volume=

    Principles of survey research: part 1: turning lemons into lemonade , author=. ACM SIGSOFT Software Engineering Notes , volume=. 2001 , publisher=

    2001

  22. [22]

    Contemporary Empirical Methods in Software Engineering , pages=

    Challenges in survey research , author=. Contemporary Empirical Methods in Software Engineering , pages=. 2020 , publisher=

    2020

  23. [23]

    2003 International Symposium on Empirical Software Engineering, 2003

    Conducting on-line surveys in software engineering , author=. 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings. , pages=. 2003 , organization=

    2003

  24. [24]

    Empirical Methods and Studies in Software Engineering: Experiences from ESERNET , pages=

    Practical experiences in the design and conduct of surveys in empirical software engineering , author=. Empirical Methods and Studies in Software Engineering: Experiences from ESERNET , pages=. 2003 , publisher=

    2003

  25. [25]

    ACM SIGSOFT Software Engineering Notes , volume=

    Principles of survey research part 2: designing a survey , author=. ACM SIGSOFT Software Engineering Notes , volume=. 2002 , publisher=

    2002

  26. [26]

    Information and Software Technology , volume=

    An empirically evaluated checklist for surveys in software engineering , author=. Information and Software Technology , volume=. 2020 , publisher=

    2020

  27. [27]

    Empirical Software Engineering , volume=

    Sampling in software engineering research: A critical review and guidelines , author=. Empirical Software Engineering , volume=. 2022 , publisher=

    2022

  28. [28]

    Proceedings of the 40th international conference on software engineering: companion proceeedings , pages=

    There is no random sampling in software engineering research , author=. Proceedings of the 40th international conference on software engineering: companion proceeedings , pages=

  29. [29]

    Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

    Sampling improvement in software engineering surveys , author=. Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , pages=

  30. [30]

    Proceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering , pages=

    Are you a real software engineer? best practices in online recruitment for software engineering studies , author=. Proceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering , pages=

  31. [31]

    Plos one , volume=

    Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA , author=. Plos one , volume=. 2023 , publisher=

    2023

  32. [32]

    Evaluation & the Health Professions , pages=

    Planning for New Threats to Online Research Data Validity: The Issue of Computer-Using Agents , author=. Evaluation & the Health Professions , pages=. 2025 , publisher=

    2025

  33. [33]

    JMIR cancer , volume=

    Strategies for the identification and prevention of survey fraud: Data analysis of a web-based survey , author=. JMIR cancer , volume=. 2021 , publisher=

    2021

  34. [34]

    Methodological Innovations , volume=

    Suspicious and fraudulent online survey participation: Introducing the REAL framework , author=. Methodological Innovations , volume=. 2021 , publisher=

    2021

  35. [35]

    People and Nature , volume=

    Addressing fraudulent responses in online surveys: Insights from a web-based participatory mapping study , author=. People and Nature , volume=. 2024 , publisher=

    2024

  36. [36]

    Journal of Experimental Political Science , volume=

    Fraud in online surveys: evidence from a nonprobability, subpopulation sample , author=. Journal of Experimental Political Science , volume=. 2023 , publisher=

    2023

  37. [37]

    Anatomical sciences education , volume=

    Survey sabotage: Insights into reducing the risk of fraudulent responses in online surveys , author=. Anatomical sciences education , volume=. 2025 , publisher=

    2025

  38. [38]

    Frontiers in Research Metrics and Analytics , volume=

    AI-powered fraud and the erosion of online survey integrity: an analysis of 31 fraud detection strategies , author=. Frontiers in Research Metrics and Analytics , volume=. 2024 , publisher=

    2024

  39. [39]

    Western Journal of Nursing Research , pages=

    Mitigating Fraud in Online Surveys: A Methodological Approach , author=. Western Journal of Nursing Research , pages=. 2026 , publisher=

    2026

  40. [40]

    Journal of Medical Internet Research , volume=

    Exploring Methods to Mitigate Fraud in Web-Based Surveys: Multicase Study Analysis , author=. Journal of Medical Internet Research , volume=. 2025 , publisher=

    2025

  41. [41]

    Proceedings of the ACM web conference 2022 , pages=

    Beyond bot detection: combating fraudulent online survey takers , author=. Proceedings of the ACM web conference 2022 , pages=

    2022

  42. [42]

    2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages=

    Do you really code? designing and evaluating screening questions for online surveys with programmers , author=. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages=. 2021 , organization=

    2021

  43. [43]

    3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering , pages=

    An Investigation on How AI-Generated Responses Affect SoftwareEngineering Surveys , author=. 3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering , pages=

  44. [44]

    arXiv preprint arXiv:2203.14695 , year=

    Recruiting software engineers on prolific , author=. arXiv preprint arXiv:2203.14695 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2201.05348 , year=

    Software engineering user study recruitment on prolific: An experience report , author=. arXiv preprint arXiv:2201.05348 , year=

    Pith/arXiv arXiv

  46. [46]

    Qualitative and quantitative methods in libraries , volume=

    Secondary data analysis: A method of which the time has come , author=. Qualitative and quantitative methods in libraries , volume=

  47. [47]

    arXiv preprint arXiv:2010.03525 , year=

    Empirical standards for software engineering research , author=. arXiv preprint arXiv:2010.03525 , year=

    arXiv 2010

  48. [48]

    IEEE Transactions on software engineering , volume=

    Qualitative methods in empirical studies of software engineering , author=. IEEE Transactions on software engineering , volume=. 1999 , publisher=

    1999

  49. [49]

    Journal of Software: Evolution and Process , volume=

    Qualitative software engineering research: Reflections and guidelines , author=. Journal of Software: Evolution and Process , volume=. 2024 , publisher=

    2024

  50. [50]

    Qualitative social work , volume=

    Conducting secondary analysis of qualitative data: Should we, can we, and how? , author=. Qualitative social work , volume=. 2019 , publisher=

    2019

  51. [51]

    Shanghai archives of psychiatry , volume=

    Secondary analysis of existing data: opportunities and implementation , author=. Shanghai archives of psychiatry , volume=

  52. [52]

    IBM SPSS Statistics 25 Step by Step , pages=

    Descriptive statistics , author=. IBM SPSS Statistics 25 Step by Step , pages=. 2018 , publisher=

    2018

  53. [53]

    The journal of positive psychology , volume=

    Thematic analysis , author=. The journal of positive psychology , volume=. 2017 , publisher=

    2017