REVIEW 2 major objections 1 minor
Coping with the delineation of emerging fields: Nanoscience and Nanotechnology as a case study
T0 review · 2 major / 1 minor · reviewed 2026-05-24 · grok-4.3
Pith's one-line read Delineating nanoscience and nanotechnology remains difficult even after experts validate quantitative results.
desk verdict Incremental case study confirming known difficulties in nanoscience field delineation via three existing methods plus expert checks, with the main limitation being thin validation of the expert gold standard. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three delineation approaches (subject category, publication level, journal level) evaluated by precision and recall against expert interviews.
What would settle it
A new set of expert interviews that produces substantially different gold-standard sets for the same nanoscience corpus would show the validation step itself is unstable.
Extended reading notes
Core claim
Applying three delineation methods at different aggregation levels to nanoscience and nanotechnology and then comparing them against expert interviews reveals that field delineation is a complicated issue at both the quantitative and the qualitative level, even when experts validate results.
Load-bearing premise
Expert interviews supply an unbiased and sufficiently complete gold standard against which the quantitative methods can be evaluated.
Editorial extensions
If this is right
- Scientometric studies of emerging fields must treat delineation as an open methodological step rather than a solved preprocessing task.
- Results obtained at different levels of aggregation (category, paper, journal) can diverge, so studies should test multiple levels.
- Precision and recall against experts remain imperfect, indicating that hybrid or iterative refinement is still needed.
- The same difficulties are expected to appear in other emerging interdisciplinary domains.
Reading between the lines
- Methods that combine automated signals with lightweight expert feedback loops could reduce the cost of repeated validation.
- Publication-level approaches might be extended by incorporating citation or co-authorship networks to capture boundary cases the current methods miss.
- Longitudinal tests on the same field over time could reveal whether delineation quality improves or degrades as the field matures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reviews literature on field delineation challenges, then applies three quantitative approaches (subject-category, publication-level, and journal-level) to delineate nanoscience and nanotechnology. It uses expert opinion interviews as a validation step, computes precision and recall for each approach against the expert-derived labels, and concludes that field delineation remains complicated at both quantitative and qualitative levels even with expert validation.
Significance. If the expert-based evaluation is robust, the work provides concrete evidence of the practical difficulties in delineating emerging interdisciplinary fields using standard scientometric techniques, which is relevant for improving methods in research evaluation and science policy studies.
major comments (2)
- [Expert opinion interviews / methods] The description of the expert interviews (mentioned in the abstract and methods) provides no details on expert selection protocol, number of experts, coverage of sub-areas within nanoscience/nanotechnology, inter-expert agreement, or how individual assessments were aggregated into a definitive gold-standard set. This is load-bearing for the central claim because the reported precision and recall values (and thus the conclusion that delineation is complicated 'even when experts validate results') rest entirely on the assumption that these interviews form a reliable, unbiased, and complete reference; without these elements the comparative evaluation cannot be verified.
- [Results / evaluation] No data tables, exclusion rules, or step-by-step computation details are supplied for how the three delineation approaches (subject-category, publication-level, journal-level) were operationalized or how precision/recall were calculated against the expert labels. This prevents assessment of whether the evidence actually supports the claim that all three approaches underperform relative to experts.
minor comments (1)
- [Abstract] The abstract could more explicitly name the three aggregation levels and the specific metrics used for comparison.
Simulated Author's Rebuttal
We thank the referee for the constructive comments, which identify important gaps in methodological transparency. We address each major comment below and will revise the manuscript accordingly.
read point-by-point responses
-
Referee: [Expert opinion interviews / methods] The description of the expert interviews (mentioned in the abstract and methods) provides no details on expert selection protocol, number of experts, coverage of sub-areas within nanoscience/nanotechnology, inter-expert agreement, or how individual assessments were aggregated into a definitive gold-standard set. This is load-bearing for the central claim because the reported precision and recall values (and thus the conclusion that delineation is complicated 'even when experts validate results') rest entirely on the assumption that these interviews form a reliable, unbiased, and complete reference; without these elements the comparative evaluation cannot be verified.
Authors: We agree that the current methods section lacks sufficient detail on the expert interviews. In the revised manuscript we will add a dedicated subsection specifying the expert selection protocol, number of experts, their coverage across sub-areas of nanoscience/nanotechnology, any inter-expert agreement measures, and the aggregation procedure used to form the gold-standard labels. This will make the basis for the precision and recall calculations verifiable. revision: yes
-
Referee: [Results / evaluation] No data tables, exclusion rules, or step-by-step computation details are supplied for how the three delineation approaches (subject-category, publication-level, and journal-level) were operationalized or how precision/recall were calculated against the expert labels. This prevents assessment of whether the evidence actually supports the claim that all three approaches underperform relative to experts.
Authors: We concur that the results section requires more explicit operational details. The revision will include data tables, the exclusion rules applied in each delineation approach, and step-by-step descriptions of how precision and recall were computed against the expert-derived labels. This will allow direct evaluation of the comparative performance of the three methods. revision: yes
Circularity Check
No significant circularity; derivation uses external expert labels and standard IR metrics
full rationale
The paper delineates nanoscience/nanotechnology via three standard quantitative approaches (subject-category, publication-level, journal-level), then evaluates them against independently collected expert interviews by computing conventional precision and recall. No step reduces a claimed prediction or result to a fitted parameter, self-defined quantity, or self-citation chain; the expert gold standard is treated as an external benchmark rather than derived from the authors' own prior definitions or equations. The central claim that delineation remains complicated even after expert validation therefore rests on an independent comparison rather than on any of the enumerated circular patterns.
Assumptions & free parameters
assumptions (1)
- domain assumption Expert opinion interviews can serve as a reliable benchmark for evaluating the accuracy of quantitative field-delineation methods.
Cite this review
Pith. "Pith review of Coping with the delineation of emerging fields: Nanoscience and Nanotechnology as a case study." pith.science (2026). https://pith.science/paper/64GMB7W4
@misc{pith2026190711888,
author = {Pith},
title = {Pith review of: Coping with the delineation of emerging fields: Nanoscience and Nanotechnology as a case study},
year = {2026},
howpublished = {\url{https://pith.science/paper/64GMB7W4}},
note = {Machine review of arXiv:1907.11888}
}
read the original abstract
Proper field delineation plays an important role in scientometric studies, although it is a tough task. Based on an emerging and interdisciplinary field, nanoscience and nanotechnology, this paper highlights the problem of field delineation. First, we review the related literature. Then, three different approaches to delineate a field of knowledge were applied at three different levels of aggregation: subject category, publication level, and journal level. Expert opinion interviews served to assess the data, and precision and recall of each approach were calculated for comparison. Our findings confirm that field delineation is a complicated issue at both the quantitative and the qualitative level, even when experts validate results.
Reviewed May 24, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.