{"id":"20b46d7c-21a0-478c-b268-972c8914a2e7","arxiv_id":"2506.00220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A curation pipeline with a shared data model, a repository, a knowledge graph, and a ChatGPT-based chatbot that lets researchers ask questions across human-robot datasets.","lead":"This paper describes a system for making human-robot interaction datasets easier to find and use: researchers curate the data with a shared description model, publish it with permanent identifiers, link the descriptions in a knowledge graph, and answer questions through a ChatGPT-powered chatbot. It is a useful prototype for the robotics data community, though its evaluation relies mostly on the curators rating their own system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation cannot support the claim of 'significantly improved' access: the 4.65–4.9 scores come from two self-interested raters with no baseline, and the only fitted contrast has a 95% CI that includes zero.","rationale":"The paper describes real engineering: seven datasets with DOIs, a structured template, a Neo4j KG, and a RAG chatbot. That part deserves credit. The problem is the evidence for the headline 'significantly improved' claim. The reader's weakest-assumption pick (rater objectivity) is real and is quoted explicitly in Section IV.A, but the more damaging gap is that there is no comparison arm at all: even an independent rater could not tell us whether the system improves on search, on plain ChatGPT, or on uncurated metadata. The reported BHM result makes this worse because it treats a credible interval that includes zero as confirmation of a positive effect. I therefore agree with the CONDITIONAL verdict and do not move it; the concern reinforces the conditions rather than changing them. The concrete test I propose would settle the improvement question directly by comparing against the natural baselines with external, blinded raters.","tokens_in":11299,"tokens_out":4474,"duration_ms":44492,"concrete_test":"Run a controlled comparison on the same 40 evaluation queries: (a) the full chatbot over the curated KG, (b) plain ChatGPT with raw data-report PDFs as RAG context, and (c) Dataverse keyword search. Use 3–5 external raters who were not involved in dataset creation or curation, blind to condition, and measure answer correctness plus task-completion time. If condition (a) does not beat the no-curation baseline by a margin whose 95% CI excludes zero, the 'significantly improved' claim is unsupported and the verdict should remain conditional at best.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the integrated system significantly improves access and understandability of human-robotics data. The supporting evidence is the expert review in Sections IV.A and V.A: four dimensions rated 4.65 to 4.9 out of 5 by raters who are the dataset creators/curators, plus one 20-minute think-aloud. Two conditions would have to hold for these ratings to carry the claim. First, raters must be able to judge objectively answers built from metadata they themselves wrote; Section IV.A says this directly. Second, high absolute scores must establish improvement over the status quo; but there is no baseline condition, no keyword-search comparison, no plain-ChatGPT comparison, and no uncurated-metadata condition. The only fitted effect in the BHM, gamma_comp = +0.01 with 95% CI [-0.015, 0.035] in Section V.A, contains zero and is nevertheless interpreted as 'confirming that structured metadata enhances precision.' That is not a valid reading of a null interval. Section III.F compounds the problem: knowledge-graph population relies on keyword/pattern-matching Python scripts with no reported validation, so any extraction noise propagates directly into the RAG answers. The load-bearing weakness is therefore not any one score but the absence of a comparison that would let the reported numbers support 'significantly improved.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a prototype system for curating, publishing, and accessing human-robot interaction (HRI) datasets according to FAIR principles. The system has three tightly coupled components: a robotics-specific data model and data report template; publication in the Texas Robotics Dataverse with persistent DOIs; and a Neo4j knowledge graph populated from repository metadata and data reports, which feeds a ChatGPT-based retrieval-augmented chatbot. The authors report on seven registered datasets with DOIs and evaluate the chatbot via an expert review by two raters (the dataset creators/curators) on four dimensions, a pilot think-aloud session, and a FAIR checklist. The main claims are that the system supports standardized curation, persistent publication, and natural-language retrieval and comparison across HRI datasets, and that access and understandability are 'significantly improved.'","tokens_in":11457,"tokens_out":3072,"duration_ms":32728,"significance":"If the system and evaluation were fully supported, this would be a useful contribution to HRI data practices: it addresses a real gap in robotics data curation, demonstrates a concrete architecture with seven DOI-assigned datasets, and candidly acknowledges limitations such as vague comparison queries failing. The engineering design is credible and the infrastructure choices (institutional repository, TACC storage, knowledge graph, RAG) are sensible. However, the evidence for the headline claim that access and understandability are 'significantly improved' is not commensurate with the claim. The ratings come from the same people who created and curated the metadata being evaluated, there is no baseline or comparison condition, and the only fitted effect in the statistical model has a credible interval that includes zero. The paper is better positioned as a system description and a set of curation guidelines than as a demonstrated improvement over existing access methods.","major_comments":[{"comment":"The headline scores (4.65–4.9 out of 5) are produced by the two dataset creators/curators rating a chatbot whose answers are drawn from metadata those same raters wrote. Section IV.A states this explicitly: 'The experts were the creators or curators of the datasets.' With no independent raters and no baseline condition, the ratings support at most that the system behaves as its builders expect; they do not support the abstract's claim that access and understandability are 'significantly improved.' The Bayesian hierarchical model only reweights the same self-reported judgments and cannot remove this circularity.","section":"Section IV.A and Section V.A"},{"comment":"The only fitted effect reported in the evaluation, gamma_comp = +0.01 with a 95% credible interval of [-0.015, 0.035], contains zero. The text interprets this as 'confirming that structured metadata enhances the model's precision,' but a null interval does not confirm an effect; it is inconclusive at best. Because this is the only quantitative contrast in the entire evaluation, the paper's quantitative case for improvement rests entirely on absolute scores with no comparison condition, such as an uncurated-metadata condition, keyword search, or a general-purpose chatbot.","section":"Section V.A, model for y_{ij}"},{"comment":"Knowledge-graph population relies on Python scripts that use pattern matching and keyword detection to parse structured JSON records and data-report fields, but the paper reports no validation of this extraction step (e.g., precision/recall against a manually labeled set, or error analysis per dataset). Since the RAG chatbot's factual grounding is inherited directly from these extracted nodes, any extraction errors propagate into all downstream answers. The paper should either provide extraction validation or soften the factual-accuracy claims in Section V.A.","section":"Section III.F"}],"minor_comments":[{"comment":"The phrase 'exploration, comparison robotics datasets' is missing a word or conjunction; it should be 'exploration and comparison of robotics datasets.'","section":"Abstract"},{"comment":"There is a typo in 'While more discover,able, many datasets published in institutional repositories'—'discover,able' should be 'discoverable.'","section":"Section I"},{"comment":"Figures 4 and 5 are referenced in the text but not described in enough detail; the reader cannot see the actual comparison answer or the Bayesian correction without reproducing the figures. Please add a few sentences explaining what each figure shows.","section":"Section IV and Section V"},{"comment":"The conclusion states 'The evaluation of our system demonstrated its effectiveness,' which is stronger than the abstract's 'suggests' and is not supported by the self-referential, baseline-free evaluation; please align the conclusion with the evidence presented.","section":"Section VI"},{"comment":"The statement 'GitHub will only hold up to 100 MB per project' is imprecise: GitHub's 100 MB limit applies to individual files, not repository-wide storage; please rephrase to avoid a factual error.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the HCRL workshop and the broader data-curation audience, but the evaluation section needs substantial work before the quantitative claims can stand. I would encourage the editor to ask for either a reframing of the paper as a system description with a clearly labeled pilot evaluation, or a comparative study with a baseline condition and independent raters. The engineering infrastructure and the seven DOI-assigned datasets are concrete strengths that should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes a working system: seven HRI datasets with DOIs, a shared data model and curation template, a Neo4j knowledge graph, and a ChatGPT RAG chatbot. That part is real and useful for the robotics-data community. What's actually new is the integration of these components into one reference architecture with domain-specific robotics classes and relationships, and the authors have shipped actual datasets and infrastructure, not just a mockup. They are also candid about known limitations — vague comparison queries fail, answers can be wordy, and the ethics section shows genuine engagement with human-subjects data.\n\nThe problem is the evaluation. Section IV.A states that the experts who rated the chatbot are the creators and curators of the datasets. That means the 4.65–4.9 scores measure the system against its own inputs. There is no baseline: no comparison with keyword search, plain ChatGPT, or uncurated metadata. The only fitted effect, gamma_comp = +0.01 with 95% CI [-0.015, 0.035], contains zero, and Section V.A.1 interprets this as confirming that structured metadata enhances precision. That is not a valid reading of a null interval. Section III.F also relies on pattern-matching scripts to populate the knowledge graph, with no reported validation, so extraction errors would propagate into every downstream answer.\n\nThese are addressable problems. External raters, an ablation without curation, a FAIR audit rubric, and release of the template, scripts, prompts, and evaluation queries would make the claims testable. The paper deserves a serious referee because the system exists, the datasets are real, and the qualitative observations from the think-aloud are plausible. But I would not accept it as is. The evaluation section needs major revision before the headline claim can stand.\n\nBottom line: send it to peer review, but ask for external evaluation and a proper baseline before publication. The paper is more valuable as a systems description than as evidence of improved accessibility.","headline":"A real and useful FAIR curation pipeline for HRI data, but the evaluation does not support the 'significantly improved' claim — the raters are the metadata authors and the only fitted effect has a CI that includes zero.","tokens_in":580,"tokens_out":926,"would_cite":true,"duration_ms":25986,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Curating human-robot datasets into one knowledge graph lets a retrieval-augmented chatbot answer cross-dataset questions, earning expert ratings of 4.65 to 4.9 out of 5.","keywords":["FAIR data principles","human-robot interaction datasets","data curation","knowledge graph","retrieval-augmented generation","natural language data access","institutional repositories","data reuse"],"falsifier":"Run the expert review's forty questions with five independent raters who have never seen the seven datasets, scoring the chatbot's answers against the published data reports; if their factual-accuracy and completeness averages fall well below 4.65–4.9, the claimed improvement in accessibility is not established.","tokens_in":10912,"feed_emoji":"🤖","tokens_out":12493,"duration_ms":124931,"temperature":0.7,"pith_summary":"This paper argues that the barrier to reusing human-robot interaction data is not a shortage of datasets but a shortage of curation standards, persistent publication, and a way to ask questions across datasets at once. The authors built a system in which researchers document datasets with a shared data-report template, publish them in an institutional repository with persistent identifiers, and have the metadata automatically loaded into a knowledge graph. A retrieval-augmented chatbot then answers natural-language questions about individual datasets or comparisons between them, using the graph structure to ground each answer. In an expert review, raters who created or curated the datasets gave the chatbot averages of 4.65, 4.9, 4.9, and 4.9 out of 5 for information retrieval, answer stability, factual accuracy, and comparison capability. If this holds, the practical payoff is that researchers can explore and compare robotics datasets by conversation instead of reading repository landing pages one by one.","feed_headline":"Robot-data chatbot scores 4.9/5 in expert test","feed_subtitle":"A knowledge graph plus retrieval-augmented ChatGPT turns seven robotics datasets into one searchable conversation.","key_machinery":"The load-bearing mechanism is a hierarchical robotics data model implemented as a knowledge graph. The model fixes a shared vocabulary of metadata classes and properties—robot type and model, sensors, control method, research method, experimental conditions, participant measures, and data post-processing—so that every curated dataset is described in the same terms. Published repository metadata and data-report fields are parsed into graph nodes and typed relationships, for example a robot model node such as 'Boston Dynamics Spot' linked to its dataset through a 'usesModel' relation. A retrieval-augmented generation (RAG) chatbot, a language model that is fed retrieved graph content and documents before answering, grounds its responses in these structures rather than in the model's pre-trained knowledge alone.","core_discovery":"The central discovery is that structured curation is what makes natural-language access to robotics data reliable. By imposing one hierarchical data model across datasets—robot model, sensors, control method, research method, experimental sessions, participant measures, and post-processing—the system turns heterogeneous repository records into uniformly connected graph nodes, so the same query can be answered consistently whether it targets one dataset or several. The chatbot's answers are not generated from general language-model knowledge; they are retrieved and synthesized from the knowledge graph and the associated data reports, which is why the paper expects factual grounding rather than hallucination. The evaluation reports that this design achieved expert ratings of 4.65 to 4.9 across the four quality dimensions.","pith_inferences":["A decisive test the paper does not run is a blind comparison between this chatbot and the same language model prompted with uncurated repository landing pages; that comparison would isolate how much of the 4.65–4.9 performance comes from the curation pipeline itself.","If this pattern generalizes, the design of curating once and then querying through a knowledge graph plus a conversational interface could transfer to other fields whose data is self-published and heterogeneous, such as ecology or mixed-methods social science.","The pattern-matching extraction step that populates the knowledge graph suggests a measurable engineering target: building a hand-labeled gold standard of data-report fields and measuring extraction precision, since every downstream answer inherits extraction errors.","The authors' own finding that vague comparison prompts produce poor answers implies the next interface improvement may be query suggestion or disambiguation rather than more training data."],"forward_implications":["A researcher can ask questions such as 'Which datasets use joystick-based teleoperation versus autonomous navigation?' and receive a synthesized answer drawn from the knowledge graph instead of a list of links to inspect one by one.","New datasets become comparable to existing ones as soon as they are curated with the same template, because the data model provides a fixed vocabulary for robot, human, and experimental attributes.","Large datasets can remain permanently accessible when small files live in the repository with persistent identifiers and large files live on scalable web-accessible storage, as done for the four-terabyte dataset in the system.","Because the chatbot's answer quality is tied to curation quality, the system creates a direct incentive for researchers to document provenance, ethics decisions, and data dictionaries at publication time.","The same pipeline is intended to extend beyond human-robot interaction to broader robotics datasets and to additional repositories, which the authors list as future work."],"supporting_citations":[{"why":"Supplies the FAIR principles that the system implements and against which its outcomes are checked.","marker":"[4]"},{"why":"Documents the scattered landscape of robotics dataset hosting that motivates the system.","marker":"[5]"},{"why":"Provides the baseline DDI metadata schema that the data model extends for robotics-specific provenance.","marker":"[15]"},{"why":"Shows a successful knowledge-graph schema from biology that motivates the graph-based integration design.","marker":"[16]"},{"why":"Demonstrates knowledge-graph infrastructure for semantic queries in another domain as a precedent for this system.","marker":"[18]"},{"why":"Is one of the seven testbed datasets registered in the system and used in the expert review.","marker":"[26]"},{"why":"Is the large-scale dataset that stresses the repository-plus-scalable-storage design and serves as a query target.","marker":"[28]"},{"why":"Supplies one of the evaluation dimensions used to rate the chatbot's information quality.","marker":"[34]"},{"why":"Supplies another of the evaluation dimensions used to rate the chatbot's information quality.","marker":"[35]"}],"fun_headline_variants":["Robot-data chatbot earns 4.9/5 from experts","Curate, Connect, Inquire: FAIR data chatbot evaluated","ChatGPT-powered tool raises robotics data access","Structured curation powers reliable data chatbot","Expert-tested system makes robotics data searchable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline scores assume that the two expert raters, who also created and curated the metadata the chatbot answers from, can judge the chatbot's answers objectively; if that objectivity is missing, the 4.65–4.9 averages validate the system against its own inputs rather than its service to outside researchers.","fun_headline_variants_meta":{"raw":{"variants":["Robot-data chatbot earns 4.9/5 from experts","Curate, Connect, Inquire: FAIR data chatbot evaluated","ChatGPT-powered tool raises robotics data access","Structured curation powers reliable data chatbot","Expert-tested system makes robotics data searchable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3793,"prompt_tokens":926,"completion_tokens":2867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2792}},"tokens_in":542,"tokens_out":2867,"duration_ms":26508,"temperature":1.0,"reasoning_tokens":2792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:11:09.900833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the expert review's forty questions with five independent raters who have never seen the seven datasets, scoring the chatbot's answers against the published data reports; if their factual-accuracy and completeness averages fall well below 4.65–4.9, the claimed improvement in accessibility is not established.","supporting_citations":[{"cited_title":"Zenodo: A Dataset Sharing Platform for Scientific Research,","cited_arxiv_id":null,"evidence_quote":"Documents the scattered landscape of robotics dataset hosting that motivates the system."},{"cited_title":"Data Documentation Initiative (DDI) Metadata Specifi- cation,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline DDI metadata schema that the data model extends for robotics-specific provenance."},{"cited_title":"KG-Hub -- Building and Exchanging Biological Knowledge Graphs","cited_arxiv_id":"2302.10800","evidence_quote":"Shows a successful knowledge-graph schema from biology that motivates the graph-based integration design."},{"cited_title":"The KnowWhereGraph Ontology","cited_arxiv_id":"2410.13948","evidence_quote":"Demonstrates knowledge-graph infrastructure for semantic queries in another domain as a precedent for this system."},{"cited_title":"Community Embedded Robotics: Vid2Real—An online video dataset about per- ceived social intelligence in human–robot encounters (Version 1) [Data set],","cited_arxiv_id":null,"evidence_quote":"Is one of the seven testbed datasets registered in the system and used in the expert review."}],"review_version":1}