The Weakness of the Scientific Assessments: A Praise of Silence

Jose Carlos Bermejo Barrera

REVIEW

The Weakness of the Scientific Assessments: A Praise of Silence

Jose Carlos Bermejo Barrera

Physics and Society · Submitted 24 Feb 2006 · 25 references

Pith's verdict: A thought-provoking essay that packages known critiques of metrics into six pseudo-axioms and a nice citation paradox, but the advertised 'contradiction' is never shown; worth a serious read, not a serious endorsement.

T0 · AI review

How this review was made: An AI referee (deepseek-v4-flash) read the paper on August 28, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 4 major objections · 5 minor · Ask about this paper · The compiled record · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv IDphysics/0602167 (version 1)

Pith ID3R2QCT62, Pith's permanent record of this paper and its review

CitationJose Carlos Bermejo Barrera. "The Weakness of the Scientific Assessments: A Praise of Silence." arXiv:physics/0602167 (2006). https://pith.science/paper/3R2QCT62

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

You should know two things about this essay. First, it has a genuinely useful insight: the observation that foundational results stop being cited because they become facts — the 'fundamental paradox' that a truly important scientist is one nobody quotes. That is a real phenomenon, and it is the best thing in the paper. Second, the essay's headline claim — that scientific assessment rests on six contradictory pseudo-axioms — is never actually demonstrated. Each pseudo-axiom is argued to be false or contestable, but falsity is not inconsistency. The closest the paper gets to a contradiction is the tension between the five-year lifespan and the view that science can forecast its future, but that is not a logical contradiction: citation half-life and short-term planning can coexist. The central claim, as worded, is unsupported even if every individual critique is accepted.

What the paper does well is packaging. The six-pseudo-axiom framing, the pairing of singularity versus anonymity and language versus silence, and the demolition of the h-index as 'a liking for symmetry' are memorable and rhetorically effective. The paper also gives credit where it is due: it draws on Gross on rhetoric, Collins on emotional energy, Bachelard on the anonymity of modern science, and Becher and Trowler on disciplinary differences. Those are the right sources for the essay's modest claims, and the essay's modest claims are largely plausible. Citations are social signals, not units of knowledge; fields do differ in publication norms; and evaluation is partly a social and political process.

The soft spots are real and load-bearing. The heterogeneity-of-knowledge premise — that knowledge is heterogeneous and therefore unquantifiable — is asserted, not demonstrated, and it reappears almost verbatim as conclusion 1. That is circular in structure. The paper also ignores the primary bibliometric and Mertonian literature: no Hirsch beyond a newspaper column, no Merton, no citation half-life studies. The 'fundamental paradox' is overstated; unquoted work is mostly negligible, and many foundational papers are heavily cited. There is also a blank figure, and at one point the paper calls the appeal to authority 'argumentum ad baculum,' which is actually appeal to force — a minor but telling error. The prose is occasionally ragged.

Who is this for? Someone thinking about why metrics feel wrong, or teaching a course on science studies, would get value from the framing. It is not a rigorous contribution to the sociology of science; it is a philosophically informed polemic. I would send it to a serious referee for a science studies journal, not desk reject it, on the strength of the citation paradox and the clear need for revision. But it needs major work to make the contradiction claim honest, and to engage the literature it attacks. My own verdict is skeptical of the central thesis, but the paper is worth engaging.

Referee report

Significance

If the central thesis were established, the paper would have considerable significance for science policy, bibliometrics, and the sociology of science: it would undermine the legitimacy of citation-based metrics such as the h-index and challenge the idea that peer review and evaluation can be value-neutral. The essay does contain some valuable and well-grounded observations: the critique of the article as a 'rhetorical unit' rather than a unit of knowledge is supported by the cited work of Gross, and the treatment of citations as social recognition signals is consistent with Collins's sociology. The discussion of field differences in publication and authorship practices (Becher and Trowler) is also pertinent. However, the paper's main claim, as stated in the abstract and introduction, is that the six pseudo-axioms are 'in contradiction,' and this claim is never demonstrated. The analysis mostly shows that each axiom is contestable or false under a particular philosophical view, not that the axioms are mutually inconsistent.

Major objections (4)

[Abstract and Introduction (pseudo-axioms list)] The central claim that the six pseudo-axioms are 'in contradiction' is never established. Each 'Weakness' section argues that a pseudo-axiom is false, unsupported, or undesirable, but falsity of individual assumptions does not imply inconsistency among them. The closest candidate, the alleged contradiction between pseudo-axiom 5 (five-year lifespan of publications) and pseudo-axiom 6 (science can forecast its future), is not a logical contradiction: a short citation half-life does not entail that all knowledge is destroyed after five years, and short-term planning can coexist with rapid obsolescence. The paper should either provide a formal statement of the contradiction or revise the central claim to 'unjustified' or 'mutually supportive but flawed.'
[Weakness of 2 and Conclusions (conclusion 1)] The argument is structurally circular. Weakness of 2 asserts that 'There is not the Science, but only the knowledge, by definition heterogeneous' as the basis for rejecting pseudo-axiom 2, and conclusion 1 restates this as 'The Science doesn't exist, but different kinds of knowledge.' The heterogeneity of knowledge is asserted, not demonstrated; the examples given (physics vs. zoology vs. philology) show diversity of methods but do not establish that no common structure, such as the use of evidence and inference, exists across fields. Because this premise is load-bearing for the claim that no universal measure can exist, the paper needs a substantive argument against unity-of-science positions, not merely an assertion.
[Weakness of 5 and 6 (p. 14-15, 'fundamental paradox')] The 'fundamental paradox of the scientific evaluation'—that a scientist is truly important only when nobody quotes his works—is asserted through a three-stage scheme but is not supported as a logical consequence of that scheme. The scheme conflates different things: an article can be unquoted because it is negligible, because it has been superseded, or because it has become a background fact; only the last case supports the paradox, and the paper gives no evidence for the frequency of that case. Foundational papers (e.g., in quantum mechanics or molecular biology) are often heavily cited long after they become canonical. The paradox is rhetorically striking but does not follow from the preceding analysis.
[Introduction (p. 1-2) and Weakness of 5 and 6 (p. 16)] The paper repeatedly attributes the weakness of assessment systems to the 'ill will' and strategic self-interest of scientists, stating that this is not due to lack of intelligence. This is an empirical psychological claim about motivations that is not supported by evidence. Even if the systems are flawed, the flaws could arise from institutional inertia, cognitive biases, or genuinely difficult design problems. The motivational claim is not necessary for the main argument and should be either supported with evidence or softened to a possibility claim.

Minor objections (5)

[Throughout] The manuscript contains numerous typographical and stylistic errors, including 'Let`s see,' '¿what is a quotation?,' 'Bachelard (1951; 1953),' and the incomplete bibliography entry 'Le matérialisme rationne,.' These should be corrected before publication.
[Weakness of 5 and 6 (p. 12-13)] The proposed impact formula I(a) = N(c) / N(t) in C(x) is introduced without defining the notation C(x) and without addressing the practical difficulty of counting the 'total number of works in a definite field.' As stated, the formula is more a rhetorical device than a usable metric.
[Weakness of 3 and 4 (p. 10)] The claim that 'authors often quote and are quoted according to definite data and findings, but never when are established general principles' is overbroad and contradicted by common citation practices in theoretical fields; it should be qualified.
[Weakness of 1 (p. 4)] The phrase 'argumentum ad baculum' is misused: an appeal to authority is not the same as an appeal to force. This is a minor terminological issue but could confuse readers with formal logic training.
[Figure 1] The text refers to 'the sketch of the figure 1' but no figure is provided or described in the manuscript. Either include the figure or remove the reference.

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Scientific assessment measures prestige, not knowledge.

The reading

This paper sets out to show that standard systems for assessing scientific work—citation counts, journal rankings, and metrics like the h-index—rest on six assumptions it calls pseudo-axioms, and that these assumptions contradict one another. The central charge is that such evaluations never measure the content of knowledge: they measure the social standing of researchers within their communities, functioning as an honors system that allocates funds and institutional control. If the argument holds, no universal quantitative parameter can measure knowledge, and decisions presented as objective assessment are at bottom social and political choices. The paper draws the consequence that there is no single thing called Science, only heterogeneous kinds of knowledge, each with its own methods, outputs, and legitimate forms of valuation.

What carries the argument

The mechanism carrying the argument is the distinction between a publication as a rhetorical unit and as a unit of knowledge, together with the treatment of the citation as a social act of recognition. A paper, the paper argues, obeys stylistic and bibliographic conventions; it is not an atom of intellectual capital, and the quality attributed to a journal is only a socially granted rank of honor. The citation, likewise, is a sign that one researcher grants authority to another, so counting citations amounts to counting acts of mutual recognition, not contributions to knowledge. On this analysis the entire apparatus of assessment—impact indexes, the h-index, five-year horizons—becomes a formalized version of an honors system, and the paper sharpens this with the paradox that a researcher is most important when nobody needs to cite him, because his results have become common facts.

Extended reading notes

Core claim

The paper's core claim is that the major scientific assessment systems are logically weak because they depend on contradictory axioms: only scientists can assess science; knowledge production is quantifiable; the publication is the unit of measure; publications can be ranked by objective criteria; published work decays within roughly five years; and science can discard its past and forecast its future. The first two pseudo-axioms are meant to clash with the heterogeneity of knowledge, the next two with the fact that papers are rhetorical units and citations are signs of mutual recognition rather than units of knowledge, and the last two with the paradox that truly fundamental work stops being cited once it becomes accepted fact. Consequently, the paper claims, evaluation of scientific knowledge in itself does not exist; what is measured is the external signs of a prestige order, and scientists who speak about science as a whole are only expressing opinions. The positive proposal is that evaluation should follow the heterogeneous forms of knowledge and curricula, with funding and rank decided by historical and ethical values rather than by a single universal scale.

Load-bearing premise

The load-bearing premise is that kinds of knowledge are radically heterogeneous and share no common internal structure, so no universal quantitative parameter can measure them.

Editorial extensions

If this is right

If assessment is an honors system, then funding and hiring decisions made on citation metrics do not track the epistemic value of research, no matter how precisely they are computed.
Because no one has authoritative competence about science as a whole, public and institutional debates about research priorities are ethical and political discussions, not technical ones.
The heterogeneity of knowledge implies that a single evaluation standard systematically favors fields with high publication output and teamwork over fields that produce books, single-author work, or slow-maturing ideas.
The five-year lifespan rule is arbitrary, and citation counts can actually invert importance: work that becomes background knowledge is cited less, not more.
Scientific futures cannot be forecast, so claims to long-term prediction made in funding proposals are strategic rhetoric rather than grounded forecasts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This suggests a testable corollary: citation trajectories of papers that later become textbook facts should follow a rise-then-fall pattern, since their content is absorbed and no longer attributed; a dataset of foundational papers could check this directly.
The argument points toward a plural-evaluation design in which assessment panels rotate among epistemic cultures and use field-relative criteria, though the paper does not spell out such a mechanism.
If the heterogeneity premise is right, the same research output would be judged differently by differently composed panels; an experiment comparing panel compositions would give this a sharp empirical handle.
Read as a policy view, the paper implies that research funding should follow an explicit value ranking, such as the values of the Declaration of Human Rights, rather than an internal scientific hierarchy—a consequence the author only sketches in the final list of ideas.

Original abstract

This article aims to show the weakness of the current scientific assessments, based on a set of contradictory pseudo-axioms. The six pseudo-axioms are deeply analysed. From the analysis are derived several conclusions. In spite of the serious efforts of the scientists to establish a ranking of honors in order to get funds or to control the academic and higher education institutions, the Science doesn't exists in itself, but different kinds of knowledge. Therefore, the scientists don't control the Science; they are mere experts in a field of knowledge, which could be expressed in many valid ways. This variety determines the existence of many models of academic curricula, based on heterogeneous ways of valuation that change along the History.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,798 input tokens.

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.08 · human participants 0.05 · controlled comparison 0.02 · randomized assignment 0.01 · states quantitative result 0.12 · compares with prior results 0.04 · mentions code or data availability 0.01.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

This article aims to show the weakness of the current scientific assessments, based on a set of contradictory pseudo-axioms. The six pseudo-axioms are deeply analysed. From the analysis are derived several conclusions. In spite of the serious efforts of the scientists to establish a ranking of honors in order to get funds or to control the academic and higher education institutions, the Science doesn't exists in itself, but different kinds of knowledge. Therefore, the scientists don't control the Science; they are mere experts in a field of knowledge, which could be expressed in many valid ways. This variety determines the existence of many models of academic curricula, based on heterogeneous ways of valuation that change along the History.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	This article aims to show the weakness of the current scientific assessments, based on a set of contradictory pseudo-axioms.	contribution 0.98	0.39	0.15	asserted 0.98
s1	The six pseudo-axioms are deeply analysed.	method 0.58	0.83	0.03	asserted 0.98
s2	From the analysis are derived several conclusions.	result 0.56	0.83	0.05	asserted 0.97
s3	In spite of the serious efforts of the scientists to establish a ranking of honors in order to get funds or to control the academic and higher education institutions, the Science doesn't exists in itself, but different kinds of knowledge.	result 0.80	0.13	0.47	asserted 1.00
s4	Therefore, the scientists don't control the Science; they are mere experts in a field of knowledge, which could be expressed in many valid ways.	interpretation 0.63	0.36	0.38	asserted 0.96
s5	This variety determines the existence of many models of academic curricula, based on heterogeneous ways of valuation that change along the History.	interpretation 0.60	0.33	0.16	asserted 1.00

What kind of paper the classifier reports this is: position or commentary 1.00 · evidence from argument 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

References

Works cited (25)

[1]

Only the scientists have the ability of assessing the development of their own subject and the overall system —institutional, political and economic— of production of the knowledge

View on Pith
[2]

The production of knowledge is a formalized process, even quantifiable, according to some parameters of universal validity

View on Pith
[3]

Just like any process of quantifying, it is essential establish a unit of measure, and that unit of meas ure must be a publication of any kind

View on Pith
[4]

The publications are arranged in order of importa nce, according to criteria easily objectivable

View on Pith
[5]

The publications have an averag e lifespan that depends on the dynamic of the fast discovering, wh ich characterizes the most of the fields of the scientific research

View on Pith
[6]

That dynamic of the fast discovering allows that the Science could do systematically without its past and forecast its own future. Let`s see the weakness of these pseudo-axioms. Weakness of 1: A Science could be defined in this way: We call Science to a limited field of the knowledge that must have, first, an object or a group of objects defining its cont...

View on Pith 1990
[7]

A scientist works inside the framewor k of a theory. That means that he accepts its principles, uses the methods established by his community and gets some results admitted by the group of scientists who share those principles, use the same methods and show them according to the usual conventions, in the approved publications or in the institutional frame...

View on Pith
[8]

A scientific revolution involves to change the principles (all of them or partly) and the methods, to get unexpected result s which could be shown in new forms, for example using other mathematical formulisms, and to split up with the ranking of honors of the scientific comm unity, because those who gather the previous scientific capital will lose most of...

View on Pith 1997
[9]

Emotional energy, necessary for making constantly the big effort of the intellectual work

View on Pith
[10]

Accumulation of cultural —or scientific— capital

View on Pith
[11]

Institutional recognition. A scientist must focus his emotional en ergy in the achieveme nt of his cultural capital and of his recognition. These two achievements are usually interrelated, although sometimes the recognition and the capital don ’t happen simultaneously. Scientists with less intellectual capital than ot hers could be recogni zed by social o...

View on Pith 1996
[12]

An article is fundamental and has a large number of quotations

View on Pith
[13]

The article is not quoted, but the au thor will be part of the history of the discipline, with a law or theorem from his name: Fermat`s theorem, Maxwell`s equations... Th e author becomes so a main character in the history of his Science

View on Pith
[14]

The discovering is so important that the author is not named, because what he said is accepted as a fact: Gravity Law and not Isaac Newton`s Law. According to this patter n, we could formulate the fundamental paradox of the scientific evaluation: A scientist is truly important when nobody quotes his works. His 15 works are not quoted because the importanc...

View on Pith
[15]

The Science doesn’t exist, but different kinds of knowledge

View on Pith
[16]

There isn’t one, but many ways of producing knowledge

View on Pith
[17]

The ways of exposing the knowledge are very different. There isn’t a unity of production of the knowledge

View on Pith
[18]

There is not only a model of academic curriculum, but many

View on Pith
[19]

The diversity of curricula demands absolutely heterogeneous ways of valuation

View on Pith
[20]

Those valuations must determine both the funding of the research and of the researchers

View on Pith
[21]

The ranking of sciences may be epistemic, but not political and economical, and the science of most epistemic rank doesn’t need more funding due to it

View on Pith
[22]

The heterogeneous production of know ledge is related to different systems of values: technical, economi cal or human —with values as the Health or the moral and political values—

View on Pith
[23]

The pyramid of values is not establis hed by the scientists, but in a given historical period the values are arranged in a certain way

View on Pith
[24]

In the present historical time, th e Science, understood as a provisional search without end, must be subordinated to the values of the Declaration of Human Rights, recognized by the democratic systems

View on Pith
[25]

The democratic systems are only an approximation to the shaping of those values, in the same way that the Science is only an approximation to the conquest of knowledge. Coda In the sixteenth century a shoemaker called Jakob Böhme lived in a village of Germany near to the town of Görlitz. Böhme didn’t speak Latin and was the firs 18 philosopher that wrote ...

View on Pith 1951

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.

[1] [1]

Only the scientists have the ability of assessing the development of their own subject and the overall system —institutional, political and economic— of production of the knowledge

View on Pith

[2] [2]

The production of knowledge is a formalized process, even quantifiable, according to some parameters of universal validity

View on Pith

[3] [3]

Just like any process of quantifying, it is essential establish a unit of measure, and that unit of meas ure must be a publication of any kind

View on Pith

[4] [4]

The publications are arranged in order of importa nce, according to criteria easily objectivable

View on Pith

[5] [5]

The publications have an averag e lifespan that depends on the dynamic of the fast discovering, wh ich characterizes the most of the fields of the scientific research

View on Pith

[6] [6]

That dynamic of the fast discovering allows that the Science could do systematically without its past and forecast its own future. Let`s see the weakness of these pseudo-axioms. Weakness of 1: A Science could be defined in this way: We call Science to a limited field of the knowledge that must have, first, an object or a group of objects defining its cont...

View on Pith 1990

[7] [7]

A scientist works inside the framewor k of a theory. That means that he accepts its principles, uses the methods established by his community and gets some results admitted by the group of scientists who share those principles, use the same methods and show them according to the usual conventions, in the approved publications or in the institutional frame...

View on Pith

[8] [8]

A scientific revolution involves to change the principles (all of them or partly) and the methods, to get unexpected result s which could be shown in new forms, for example using other mathematical formulisms, and to split up with the ranking of honors of the scientific comm unity, because those who gather the previous scientific capital will lose most of...

View on Pith 1997

[9] [9]

Emotional energy, necessary for making constantly the big effort of the intellectual work

View on Pith

[10] [10]

Accumulation of cultural —or scientific— capital

View on Pith

[11] [11]

Institutional recognition. A scientist must focus his emotional en ergy in the achieveme nt of his cultural capital and of his recognition. These two achievements are usually interrelated, although sometimes the recognition and the capital don ’t happen simultaneously. Scientists with less intellectual capital than ot hers could be recogni zed by social o...

View on Pith 1996

[12] [12]

An article is fundamental and has a large number of quotations

View on Pith

[13] [13]

The article is not quoted, but the au thor will be part of the history of the discipline, with a law or theorem from his name: Fermat`s theorem, Maxwell`s equations... Th e author becomes so a main character in the history of his Science

View on Pith

[14] [14]

The discovering is so important that the author is not named, because what he said is accepted as a fact: Gravity Law and not Isaac Newton`s Law. According to this patter n, we could formulate the fundamental paradox of the scientific evaluation: A scientist is truly important when nobody quotes his works. His 15 works are not quoted because the importanc...

View on Pith

[15] [15]

The Science doesn’t exist, but different kinds of knowledge

View on Pith

[16] [16]

There isn’t one, but many ways of producing knowledge

View on Pith

[17] [17]

The ways of exposing the knowledge are very different. There isn’t a unity of production of the knowledge

View on Pith

[18] [18]

There is not only a model of academic curriculum, but many

View on Pith

[19] [19]

The diversity of curricula demands absolutely heterogeneous ways of valuation

View on Pith

[20] [20]

Those valuations must determine both the funding of the research and of the researchers

View on Pith

[21] [21]

The ranking of sciences may be epistemic, but not political and economical, and the science of most epistemic rank doesn’t need more funding due to it

View on Pith

[22] [22]

The heterogeneous production of know ledge is related to different systems of values: technical, economi cal or human —with values as the Health or the moral and political values—

View on Pith

[23] [23]

The pyramid of values is not establis hed by the scientists, but in a given historical period the values are arranged in a certain way

View on Pith

[24] [24]

In the present historical time, th e Science, understood as a provisional search without end, must be subordinated to the values of the Declaration of Human Rights, recognized by the democratic systems

View on Pith

[25] [25]

The democratic systems are only an approximation to the shaping of those values, in the same way that the Science is only an approximation to the conquest of knowledge. Coda In the sixteenth century a shoemaker called Jakob Böhme lived in a village of Germany near to the town of Görlitz. Böhme didn’t speak Latin and was the firs 18 philosopher that wrote ...

View on Pith 1951