Pith. sign in

REVIEW 3 major objections 4 minor 12 references

The Data Dilemma: Authors' Intentions and Recognition of Research Data in Educational Technology Research

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On the basis of DELFI 2024 submissions and proceedings, this paper claims that many educational technology researchers do not recognize software and qualitative materials as research data, leading them to report 'no data to publish' even…

desk verdict A modest, honest descriptive study; the 35/56 mismatch is new, but the 'lack of awareness' claim needs a transparent codebook and softer wording. read the letter →

arxiv 2506.04954 v1 pith:ST6JH6J3 submitted 2025-06-05 cs.CY

Sandra Schulz , Natalie Kiesler This is my paper
classification cs.CY
keywords openscienceresearchdatamanagementpublicationDELFIconferenceeducationaltechnologyqualitativesoftwareartifactsFAIRprinciples
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that educational technology researchers systematically under-recognize their own research data, especially software and qualitative materials, when asked whether they have data to share. The authors compare authors' self-reports from the DELFI 2024 submission system with their own content analysis of the submitted papers. In 35 of 56 submissions tagged 'no data to publish,' they identify research data that fits the definition of the German Research Foundation: qualitative data in 20 cases, software in 19, and quantitative data in 8. If correct, this result indicates a community-wide awareness gap that suppresses data sharing and reuse, and it implies a need for targeted training, guidelines, and conference policies.

What carries the argument

The analytical engine is a two-step comparison. First, the EasyChair submission metadata records each author team's self-declared data status. Second, the authors re-read every 'no data' submission, applying the German Research Foundation's definition of research data—which includes texts, survey and observation data, questionnaires, compilations, simulations, and software required to create or process data—plus two operational rules: the data had to be explicitly linked in the submission, and only data the authors gathered or developed themselves counted. The mismatch between the self-report and the researchers' classification yields the recognition gap, while a separate analysis of the accepted proceedings combines data type (qualitative, quantitative, software) with storage location to map where research data is actually published.

What would settle it

A direct survey of the 56 author teams tagged 'no data to publish,' showing them excerpts from their own submissions and asking whether each artifact (a prototype, an interview guide, a survey instrument) counts as research data, would settle the claim: if most teams answer yes and say they always regarded them as data but declined to publish for ethics, time, or policy reasons, the recognition-gap interpretation would collapse in favor of explanation by other barriers.

Watch

Extended reading notes

Core claim

The central discovery is a recognition gap: at DELFI 2024, 45% of all submissions (56 of 125) declared 'no data to publish,' yet a content analysis of those 56 submissions identified research data in 35 of them—qualitative data in 20, software in 19, and quantitative data in 8. The paper also maps the data landscape in the accepted proceedings, finding that only 17 of 55 contributions shared openly available data, mostly software (11), followed by qualitative (6) and quantitative (3) datasets, published primarily on GitHub/GitLab or OSF. The authors interpret the mismatch as evidence of limited awareness of research data definitions, alongside other potential barriers such as complexity, missing guidelines, and anonymization concerns. They further report that three shared datasets were inaccessible due to broken or expired links, which they read as a threat to the FAIR principles that the community aspires to follow.

Load-bearing premise

The conclusion rests on the assumption that the data the reviewers identified by re-reading the papers really is research data under the applied definition, and that an author checking 'no data to publish' means the author did not recognize that data rather than choosing not to share it for ethical, practical, or other reasons.

Editorial extensions

If this is right

  • If the recognition gap is real, journals and conferences cannot rely on author self-reports to measure data availability or to enforce data policies.
  • The under-recognition of software helps explain the low data-sharing rates observed in earlier DELFI studies, particularly for demo and tool papers where prototypes are a core output.
  • Qualitative data, including interview guides, teaching materials, and course designs, would remain unpublished unless the community explicitly values them as citable research data.
  • Guidelines and research data management training should be embedded in educational technology curricula, starting during study programs.
  • The prevalence of broken hyperlinks suggests that repositories issuing persistent identifiers, such as DOIs, should be preferred over cloud storage and shortened links.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • By extension, the same recognition gap likely exists in other applied computing fields that cross disciplinary boundaries, where authors trained in one tradition may not see their instruments as data.
  • A testable extension would prompt authors with examples from their own submissions; if the gap persists even when authors are shown concrete artifacts, the problem is definitional awareness, whereas if it disappears, the earlier gap may reflect survey fatigue or interpretation of the question.
  • Because the authors themselves concede that non-publication may stem from reasons other than non-recognition, the 'lack of awareness' framing should be read as one plausible mechanism rather than a proven one; a follow-up survey could disaggregate non-recognition from unwillingness or ethical concerns.
  • Conference data policies that make a data statement mandatory may reveal more unrecognized data, but without training, authors may simply write 'no data' out of habit—so policy alone may not move sharing rates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper examines whether DELFI 2024 authors recognized the research data in their own submissions and where published research data are stored. The authors use EasyChair metadata for all 125 submissions and manually review the 56 submissions tagged as having "no data to publish," reporting that 35 of them actually contained research data, especially qualitative data (N = 20) and software (N = 19). They also analyze the 55 contributions in the DELFI 2024 proceedings, finding 17 submissions with openly available data, most frequently published on GitHub/GitLab and OSF. The paper interprets the mismatch as evidence of a prevailing lack of awareness of research data among EdTec researchers and recommends guidelines, recognition, and training.

Significance. If the classification is valid, this study provides a useful, concrete quantification of a real gap in EdTec research data sharing and reuse. The paper's strengths are its use of fresh conference submission data, transparent numerical reporting of the counts, and grounding in external definitions such as the DFG definition of research data. The central limitation is that the load-bearing outcome, 35 of 56 "no data" submissions containing unrecognized research data, depends entirely on the authors' own content classification, which is not accompanied by a public codebook, an inter-rater reliability statistic, or a list of judged submissions. The abstract's phrase "indicating a prevailing lack of awareness" also reaches beyond what the data alone establish, as the paper itself acknowledges in the Limitations section. The topic is important for the DELFI community and for research data management training, but the central claim needs stronger methodological support before it can be fully accepted.

major comments (3)
  1. [3 Methodology] The manuscript does not report any inter-rater reliability statistic, a public codebook, or a list of the 56 judged submissions, yet RQ1's central count of 35 overlooked data cases rests entirely on the authors' own application of the Section 3 criteria. Please report the coding instruments, provide reliability information (e.g., independent coding of a subset with a kappa statistic), and include an appendix or supplementary material that lists or illustrates how each category was assigned. Without this, readers cannot determine whether the 35/56 mismatch would survive a different, equally defensible reading of the DFG definition.
  2. [4.1 Results] The examples of 'overseen' qualitative data include 'teaching concepts, teaching materials (e.g., in Moodle courses), lecture notes, exercises, or ChatGPT promotions.' Under the DFG definition quoted in Section 1, research data are generated during the research process or are its result; teaching artifacts that are not part of a reported empirical study do not automatically qualify as research data. The authors should clarify, ideally with concrete examples from the judged submissions, how these artifacts satisfied the criteria of being 'explicitly linked' and 'gathered or developed by the authors,' and how they distinguished study-generated materials from ordinary course materials. This distinction is load-bearing because removing debatable cases from the 35 could change the conclusion.
  3. [Abstract; 5 Discussion] The abstract's phrase 'indicating a prevailing lack of awareness, and other potential barriers' presents an interpretive reading of the mismatch, but the study only shows that authors' self-reports disagreed with the authors' classification. The Limitations paragraph correctly notes that authors 'do not necessarily lack awareness' and may have chosen not to publish data for other reasons (e.g., consent, anonymization, effort, or misunderstanding of the EasyChair options). The Discussion and conclusions should be rephrased so the primary reported result is the mismatch itself, with lack of awareness presented as one possible explanation, or the authors should add direct evidence about authors' reasons, such as a follow-up survey of the DELFI 2024 authors.
minor comments (4)
  1. [4.1 Results] In the first bullet of Section 4.1, 'ChatGPT promotions' should likely read 'ChatGPT prompts'; as printed, it is unclear what data are meant.
  2. [References; 5 Discussion] The reference list and Section 2 use 'Melkamu Jate and Striewe (2023),' but Section 5 cites 'Melkamu and Striewe (2023)'; please standardize the author name.
  3. [4.2 Results] In Section 4.2, the text and Table 1 should state explicitly that the 20 entries are type occurrences across 17 submissions, so the reader does not misread the total as 20 separate publications.
  4. [3 Methodology] The criterion 'only data authors gathered or developed themselves' would benefit from a clarification of how existing instruments used in a study are treated when the authors collected the responses; the current wording could be read either way.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central finding is a fresh empirical comparison against an externally defined benchmark (DFG research data definition), not a reduction of the conclusion into its own inputs.

full rationale

The paper's central claim is that 35 of 56 DELFI 2024 submissions tagged 'no data to publish' actually contained research data, especially qualitative data and software. This is derived by comparing authors' EasyChair self-reports with the authors' own content analysis of the submissions. The benchmark for what counts as research data is not derived from the authors' responses; it is the DFG definition quoted in Section 1, which explicitly includes texts, survey data, audiovisual information, and software required to create or process research data. The operational criteria in Section 3—'the data had to be explicitly linked in the submission' and 'we only recognized data that authors had gathered or developed themselves'—are stated coding rules, not fitted parameters or renamings of the target conclusion. The inference from mismatch to 'lack of awareness' is an interpretive step, and the paper itself acknowledges in the Limitations that other reasons may explain why authors did not indicate data; this is an honest uncertainty, not a circular reduction. The self-citations (Kiesler, Röpke, et al. 2024; Kiesler and Schiffner 2022; Schulz and Kiesler 2024) provide background, prior findings, and the proceedings as a data source, but none of them is load-bearing for the new empirical result. No equation, fitted parameter, or self-referential theorem forces the conclusion. Concerns about the validity or over-inclusiveness of the content classification are correctness risks, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a definitional choice about what counts as research data (axiom 1), on trust in self-reported metadata (axiom 2), and on the authors' own operational rules (axiom 3). No free parameters or invented entities are involved.

assumptions (3)
  • domain assumption The DFG definition of research data, which includes measurement data, texts, surveys, observation data, and software required to create or process research data, is the appropriate standard for judging whether artifacts in submissions count as research data.
    Invoked in Section 1 and applied throughout the analysis. If a narrower definition were used, many of the 'overlooked' qualitative items (e.g., teaching materials, lecture notes) would not count as research data.
  • domain assumption Authors' EasyChair answers accurately reflect their actual beliefs about having data to share, and no data was omitted due to system or user error.
    The entire RQ1 comparison treats these self-reports as the baseline. The authors themselves note in the Limitations that other reasons, such as privacy or effort, may explain authors' choices.
  • ad hoc to paper The operational criteria defined in Section 3 ('data had to be explicitly linked in the submission' and 'only data authors gathered or developed themselves') are sufficient to identify research data in the submissions.
    These criteria were introduced by the authors for this study and are not derived from a standard method. Their application determines the 35/56 count, which is the load-bearing number for the central claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Data Dilemma: Authors' Intentions and Recognition of Research Data in Educational Technology Research." pith.science (2026). https://pith.science/paper/ST6JH6J3

@misc{pith2026250604954,
  author       = {Pith},
  title        = {Pith review of: The Data Dilemma: Authors' Intentions and Recognition of Research Data in Educational Technology Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ST6JH6J3}},
  note         = {Machine review of arXiv:2506.04954}
}
read the original abstract

Educational Technology (EdTec) research is conducted by multiple disciplines, some of which annually meet at the DELFI conference. Due to the heterogeneity of involved researchers and communities, it is our goal to identify categories of research data overseen in the context of EdTec research. Therefore, we analyze the author's perspective provided via EasyChair where authors specified whether they had research data to share. We compared this information with an analysis of the submitted articles and the contained research data. We found that not all research data was recognized as such by the authors, especially software and qualitative data, indicating a prevailing lack of awareness, and other potential barriers. In addition, we analyze the 2024 DELFI proceedings to learn what kind of data was subject to research, and where it is published. This work has implications for training future generations of EdTec researchers. It further stresses the need for guidelines and recognition of research data publications (particularly software, and qualitative data).

Figures

Figures reproduced from arXiv: 2506.04954 by the authors.

Figure 1
Figure 1. Authors’ Perspective on Publication of Their Research Data data that seemed to be largely overseen was qualitative data (N = 20). Similarly, software was often (N = 19) not categorized as research data. Quantitative data was also overlooked (N = 8). In the next paragraphs we will dive deeper into the concrete data kinds of overseen data. • Qualitative data: Examples for some common research data found are interview … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

  1. [1]

    Train -the-trainer concept on re - search data management

    Biernacka, Katarzyna, Maik Bierwirth, et al. (2020). “Train -the-trainer concept on re - search data management”. In: Zenodo https://doi. org/10.5281/ZENODO 4071471. Biernacka, Katarzyna, Adrian Mulligan, et al. (Aug. 2023). Research Data Sharing and Reuse

    doi:10.5281/zenodo 2020

  2. [2]

    Lecturers' perspectives on the integration of research data management into teacher training programmes

    DOI: 10.5281/zenodo.8010617. Schneider, Jan, Bibeg Limbu, and Natalie Kiesler (2025). “Of house of cards and air castles, a deep dive into the fertile fields of educational technologies and technology enhanced learning”. In: Journal of Computing in Higher Education, pp. 1–53. DOI: 10.1007/s12528-025-09450-8. Schulz, Sandra and Juliane Jacob (2025). Lectur...

    work page Pith review arXiv doi:10.48550/arxiv.2505.21704 2025

  3. [4]

    Where’s the Data? Finding and Reusing Datasets in Computing Education

    Ed. by Sandra Schulz and Natalie Kiesler. Gesellschaft f u¨ r Informatik eV, pp. 19–21. Kiesler, Natalie, John Impagliazzo, et al. (2024). “Where’s the Data? Finding and Reusing Datasets in Computing Education”. In: Working Group Reports on 2023 ACM Confer- ence on Global Computing Education. CompEd

    work page 2024

  4. [9]

    DOI: 10.1038/sdata.2016.18

    doi:10.1038/sdata.2016.18 2016

  5. [10]

    Forschungsdaten in der Bildungstechnologie: Wor u¨ber sprechen wir und was ist wann relevant?

    Gesellschaft f u¨ r Informatik e.V. DOI: 10.18420/delfi2024_58. Striewe, Michael (2024). “Forschungsdaten in der Bildungstechnologie: Wor u¨ber sprechen wir und was ist wann relevant?” In: Proceedings of DELFI

    work page doi:10.18420/delfi2024_58 2024

  6. [11]

    The FAIR Guiding Principles for scientific data man - agement and stewardship

    Ed. by Sandra Schulz and Natalie Kiesler. Gesellschaft f u¨ r Informatik e.V. DOI: 10.18420/delfi2024_44. Wilkinson, Mark D. et al. (2016). “The FAIR Guiding Principles for scientific data man - agement and stewardship”. In: Scientific Data 3, p

    work page doi:10.18420/delfi2024_44 2016

  7. [20]

    “Die digitale Forschungswelt

    Fachta- gung Bildungstechnologien (DELFI). Ed. by Peter A. Henning, Michael Striewe, and Matthias W¨olfel. Bonn: Gesellschaft f u¨ r Informatik e.V., pp. 201–206. DOI: 10.18420/ delfi2022-034. Kindling, Maxi and Peter Schirmbacher (2013). ““Die digitale Forschungswelt” als Gegen- stand der Forschung/Research on Digital Research/Recherche dans la domaine d...

    work page 2013

  8. [21]

    Bonn: Gesellschaft f u¨ r Informatik e.V., pp

    Fachtagung Bildungstechnologien (DELFI). Bonn: Gesellschaft f u¨ r Informatik e.V., pp. 167–172. ISBN: 978-3-88579-732-6. DOI: 10.18420/ delfi2023-27. Petersen, Britta et al. (June 2023). Lernzielmatrix zum Themenbereich Forschungsdaten- management (FDM) f u¨r die Zielgruppen Studierende, PhDs und Data Stewards . Ver- sion

    work page 2023

Show all 12 references
  1. [22]

    On the Lack of Recognition of Software Artifacts and IT Infrastructure in Educational Technology Research

    Fachtagung Bildungstechnologien (DELFI). Ed. by Sandra Schulz and Natalie Kiesler. Bonn: Gesellschaft f u¨ r Informatik e.V., pp. 251–265. DOI: 10.18420/ delfi2024_22. Kiesler, Natalie and Daniel Schiffner (2022). “On the Lack of Recognition of Software Artifacts and IT Infras...

    2022

  2. [60]

    Towards Open Science at the DELFI Confer- ence

    DOI: 10.1145/3598579.3689378. Kiesler, Natalie, Re n´e R¨opke, et al. (2024). “Towards Open Science at the DELFI Confer- ence”. In:

    2024

  3. [2020]

    Confidence vs Insight: Big and Rich Data in Computing Education Research

    Online. Mendeley Data, V1. DOI: 10.17632/nr9n75cpv2.1. Biernacka, Katarzyna and Sandra Schulz (2022). Forschungsdatenmanagement in der In- formatik. Logos Verlag. DOI: 10.30819/5490. Brown, Neil C. C. and Mark Guzdial (2024). “Confidence vs Insight: Big and Rich Data in Comput...

    2022 doi

  4. [2024]

    The interplay between rich and big data in programming edu- cation research

    Portland, OR, USA: ACM, pp. 158–164. DOI: 10.1145/3626252.3630813. European Commission and Directorate-General for Research and Innovation (2018). Cost- benefit analysis for FAIR research data – Cost of not having FAIR research data. Publications Office. DOI: 10.2777/02999. Ge...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.