REVIEW 3 major objections 5 minor 10 references
The Waters We Swim In: Replicability and the Evolution of Scientific Norms
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This essay argues that replicability and objectivity are evolving social norms, so addressing the replication crisis in physics requires cultural change, not just stricter methodology.
desk verdict A clear, well-sourced philosophical synthesis on scientific norms, but its central premise that physics needs a cultural fix for a replication crisis is asserted rather than measured; treat it as a useful commentary, not a research result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The conceptual machinery is the application of emergence to scientific knowledge itself: just as the collective behavior of many particles gives rise to properties not predictable from the laws governing individual particles, the collective practices of a scientific community give rise to what counts as valid, reproducible knowledge. The paper combines this with a historical account of objectivity as shifting through distinct ideals—truth-to-nature, mechanical objectivity, and trained judgement—and with a social model of objectivity that locates it in community critical interaction and diversity of perspectives. These concepts do the work of moving the replication crisis from a technical problem of methodology to a cultural problem of norms and values.
What would settle it
A large-scale, pre-registered audit that attempts to independently reproduce a random sample of recently published physics results across subfields—using the original raw data and analysis code—and finds a high success rate would undercut the claim that physics faces an urgent replication crisis.
Extended reading notes
Core claim
The paper's central claim is that the very notions of replicability and objectivity are emergent and historically changing properties, produced by the collective actions and interactions of scientific communities, rather than timeless criteria that individual scientists either satisfy or violate. It follows that the replication crisis is a sociological and cultural problem: current incentives have created norms that undervalue replication, critique, and transparency. Because scientists themselves create and reproduce these norms through their everyday choices, the norms can be reshaped. The paper does not offer a new technical protocol; instead it argues that a durable response requires redefining what replication and self-correction mean under current pressures, rewarding replication work, reimagining the scientific paper as a living document, and building multidisciplinary spaces for reflection.
Load-bearing premise
The load-bearing premise is that physics actually has a replication problem serious enough to warrant a cultural overhaul, supported in the paper by high-profile retractions and a recent reproducibility report rather than by measured replication rates across physics subfields.
Editorial extensions
If this is right
- Because replicability is an evolving norm, technical checklists and stricter statistics are necessary but insufficient; without cultural change they will not restore trust in physics.
- If norms are shaped by incentives, then changes in funding priorities, hiring criteria, and publication practices are direct levers for improving replicability.
- If objectivity depends on community critical interaction, then increasing diversity of viewpoints and protecting dissent within research groups and review should improve reliability.
- If trained judgement is unavoidable in data selection and analysis, transparent communication of those judgement calls becomes essential to prevent selective reporting.
- If scientific papers are treated as closed, static units of knowledge, then the rise of digital tools will not by itself produce more transparent or revisable science; norms must change as well.
Reading between the lines
- Editor's inference: The cultural diagnosis yields a measurable prediction—communities, journals, or subfields that explicitly reward replication and critical exchange should show higher independent-success rates, a claim that could be tested with a large replication audit.
- Editor's inference: If the argument is right, format changes such as living papers or open data will fail unless they are coupled to credit and incentive reform; otherwise they will be adopted superficially without changing practice.
- Editor's inference: The historical account of objectivity suggests that the field may need context-specific replication standards rather than one universal criterion, since exact duplication is often impossible in complex experimental settings.
- Editor's inference: The paper's call for collaboration with the humanities is itself a testable proposal: structured exchanges between physicists and historians or philosophers could be evaluated by whether they generate new norms and practices within physics groups.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This perspective essay argues that the replication crisis is not merely a technical problem in physics but a symptom of culturally and historically contingent norms. The authors draw on Anderson's concept of emergence, Ziman's sociology of science, Daston and Galison's history of objectivity, Longino's social epistemology, and Oreskes's work to support the view that replicability and objectivity evolve with community practices. They conclude that addressing the replication crisis requires cultivating a culture that values replication, critique, and transparency as much as discovery, and they recommend multidisciplinary collaboration between physicists and humanities/social-science scholars.
Significance. The essay's main contribution is a clear, accessible synthesis of historical and philosophical scholarship applied to the replication debate in physics. Its strength is the explicit connection between Daston and Galison's historical account of objectivity and Longino's social epistemology, which gives physicists a vocabulary for understanding scientific norms as changeable rather than fixed. The paper does not present new empirical data or machine-checked proofs, but it makes a falsifiable and important claim: if physics reproducibility is already high, the urgency of a cultural overhaul diminishes. The proposal for multidisciplinary collaboration is concrete enough to seed productive debate, and the essay productively bridges physics with philosophy and history of science.
major comments (3)
- [Introduction (first paragraph)] The load-bearing premise that physics is at risk of a replication crisis is asserted rather than demonstrated. The sentence '...their relevance is not limited to a few 'problem' areas' is supported only by refs 3 and 4, and no replication or reproducibility rates for physics subfields are reported. The high-profile retractions in high-Tc superconductivity and Majorana physics are anecdotal, documenting fraud or error in specific cases rather than systemic prevalence. Because the paper's conclusion—that addressing the replication crisis requires cultural change—depends on this premise, the essay should either present quantitative evidence (e.g., from the arXiv reproducibility report, ref 4) or explicitly reframe the argument as precautionary rather than diagnostic.
- [Scientific knowledge as an emergent property] The analogy between Anderson's physical emergence and Ziman's account of scientific knowledge is asserted rather than argued. The phrase 'In a similar manner, argued Ziman' does not specify in what sense social norms are emergent or how spontaneous symmetry breaking maps onto social epistemology. Since the paper's central claim is that replicability and objectivity are 'emergent and evolving,' this analogy is load-bearing. The authors should add a substantive explanation of the notion of emergence at work and its limits, rather than relying on rhetorical similarity.
- [Introduction (second paragraph)] The dismissal of the self-correction view with 'But think again' is a rhetorical move rather than an argument. If the essay intends to challenge the claim that physics is self-correcting, it should engage with the relevant literature, including the von Hippel reference (ref 2), which documents a replication crisis in early 20th-century chemistry and the conditions under which self-correction occurred. Without such engagement, the paper does not justify overriding the common physicist viewpoint it explicitly acknowledges, and the urgency of its prescription rests on an unexamined contrast with an underexplored opposing position.
minor comments (5)
- [Title/first line] The first line of the full text contains 'Evolu3on' and 'Scien3fic' in place of 'Evolution' and 'Scientific'; this appears to be a typographical artifact and should be corrected.
- [Where do we go from here?] The sentence 'A step forward in this discussion requires accepting that that the knowledge we produce...' contains a doubled 'that' and should be edited to remove the repetition.
- [References] Reference 9 (Oreskes 2004) is a paper on scientific consensus about climate change; it does not directly support the claim that scientific knowledge is 'inherently social.' A more apt citation to Oreskes's work on trust in science or the social nature of scientific knowledge would strengthen the argument.
- [Definitions] The essay uses 'replicability' and 'reproducibility' interchangeably without defining them; standard definitions (e.g., new experiment versus reanalysis of the same data) would sharpen the argument and avoid confusion in the physics audience.
- [Reference 4] The author list of reference 4 appears incomplete or incorrectly formatted (e.g., 'Quader, B. S. K. F.' with an unclear sequence); please verify the full author list and the arXiv identifier against the original report.
Circularity Check
No circularity: the essay is an interpretive argument that borrows its premises from external scholarship; it performs no numerical derivation and fits no parameters, so there is no reduction of outputs to inputs.
full rationale
This paper is a perspective/essay, not a quantitative derivation. It contains no equations, no fitted parameters, and no quantity defined in terms of another so as to make a later claim true by construction. The central claim — that replicability and objectivity are historically and socially shaped — is supported by citations to Ziman, Daston and Galison, Longino, and Oreskes, which are external sources rather than self-citations; none of the cited works is by the present authors, and none of the cited results is presupposed by the authors' own prior work. The paper's warning that physics could face a replication crisis is asserted with anecdotal retractions and a recent arXiv report, but a weak or unquantified premise is a correctness/evidence concern, not circularity: the conclusion does not define or fit that premise into being. The absence of measured reproduction rates across physics subfields leaves the urgency of the cultural prescription open to challenge, but the argument remains an interpretation of external historical and philosophical sources rather than a self-referential derivation. No step in the paper reduces to its own input, and no fitted value is renamed as a prediction. Score 0 is therefore appropriate.
Assumptions & free parameters
assumptions (5)
- domain assumption Scientific knowledge is an emergent property of community norms and practices, analogous to physical emergence.
- domain assumption Objectivity and replicability are historically contingent and evolve over time.
- domain assumption Objectivity is achieved through community-level critical interaction, not individual detachment.
- domain assumption External pressures such as funding, metrics, and commercialization currently distort research norms.
- ad hoc to paper Physics is at risk of a replication crisis.
Cite this review
Pith. "Pith review of The Waters We Swim In: Replicability and the Evolution of Scientific Norms." pith.science (2026). https://pith.science/paper/RIWTGWXA
@misc{pith2026250105788,
author = {Pith},
title = {Pith review of: The Waters We Swim In: Replicability and the Evolution of Scientific Norms},
year = {2026},
howpublished = {\url{https://pith.science/paper/RIWTGWXA}},
note = {Machine review of arXiv:2501.05788}
}
read the original abstract
In recent years, a series of high-profile retractions and fraud cases have arisen in physics, sparking a conversation about research integrity and replicability. Here, we discuss how the practice of science is shaped by the social and political context in which it operates. Reflection on our norms and values could provide a route to create community-driven safeguards that respond to the changing demands in which our research occurs. We propose that collaborations between physicists, philosophers, social scientists, and historians of science could facilitate these reflections and provide new ideas for social science and humanities colleagues.
Reference graph
Works this paper leans on
-
[1]
A wave of retractions is shaking physics
Chen, S. A wave of retractions is shaking physics. MIT technology review (2024). at <https://www.technologyreview.com/2024/05/15/1092535/a-wave-of-retractions-is-shaking-physics/>
work page 2024
-
[2]
Hippel, P. von. When Does Science Self-Correct? Lessons from a Replication Crisis in Early 20th Century Chemistry. The Good Science Project (2023). at <https://goodscience.substack.com/p/when-does-science-self-correct-lessons>
work page 2023
-
[3]
How can we make science more rigorous? Nat
Anirban, A. How can we make science more rigorous? Nat. Rev. Phys. 5, 498–499 (2023)
work page 2023
-
[4]
Akrap, A., Bordelon, D., Chatterjee, S., Dahlberg, E. D., Devaty, R. P., Frolov, S. M., Gould, C., Greene, L. H., Guchhait, S., Hamlin, J. J., Hunt, B. M., Jardine, M. J. A., Kayyalha, M., Kurchin, R. C., Kozii, V., Legg, H. F., Mazin, I. I., Mourik, V., Özgüler, A. B., Peñuela-Parra, J., Seradjeh, B., Quader, B. S. K. F. & Zwolak, J. P. Report on Reprodu...
-
[5]
Anderson, P. W. More Is Different. Science 177, 393–396 (1972)
work page 1972
-
[6]
Is science losing its objectivity? Nature 382, 751–754 (1996)
Ziman, J. Is science losing its objectivity? Nature 382, 751–754 (1996)
work page 1996
-
[7]
Daston, L. & Galison, P. Objectivity. 309–362 (2021). doi:10.2307/j.ctv1c9hq4d.11
- [8]
Show all 10 references
-
[9]
The Scientific Consensus on Climate Change
Oreskes, N. The Scientific Consensus on Climate Change. Science 306, 1686–1686 (2004)
2004
-
[10]
de, Thresher, A
Swart, J. de, Thresher, A. C. & Argüelles, C. A. The humanities can help make physics greener. Nat. Rev. Phys. 6, 404–405 (2024)
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.