QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing

Anya Belz

REVIEW 3 major objections 5 minor 29 references

QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read QRA++ measures reproducibility as a continuous degree, comparable across NLP studies, grounded in how similar experiments are.

desk verdict A useful synthesis of reproducibility assessment for NLP, but the empirical demonstration is overclaimed and the Table 6 example is not consistent with the framework's own property set. read the letter →

arxiv 2505.17043 v1 pith:6ENAHGX6 submitted 2025-05-13 cs.CL cs.AI

Anya Belz This is my paper · ORCID

classification cs.CLcs.AI

keywords reproducibilitynaturallanguageprocessingevaluationmetrologymeasurementprecisioncoefficientofvariationexperimentpropertiesdegree

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reproducibility in NLP should be assessed as a matter of degree rather than a binary success/failure, and presents QRA++, a framework that produces continuous-valued reproducibility scores at three levels of granularity: individual systems, quality criteria, and whole studies. The framework grounds expectations about reproducibility in the degree of similarity between experiments, using an explicit set of experiment properties as measurement conditions. Because the measures are unitless and defined uniformly across studies, QRA++ makes reproducibility assessments directly comparable across different papers and shared tasks. Applied to three sets of comparable experiments, QRA++ shows that reproducibility is better when experiments share the same test data and seeds, and that system type and evaluation method also drive differences. If adopted, the framework would let the field move from narrative judgments to quantitative, diagnostic reproducibility reporting.

What carries the argument

The load-bearing object is the equation that identifies reproducibility with the precision of a set of measured values: $R(M_1,\dots,M_n) := \mathrm{Precision}(v_1,\dots,v_n)$, where each measurement $M_i : (m,O,t_i,C_i) \mapsto v_i$ carries measurand, object, time, and measurement conditions. In NLP terms, objects are systems, measurands are evaluation measures, and conditions are experiment properties. QRA++ supplies a curated checklist of such properties (task, input and output type, test dataset, metric implementation, run-time environment, and several human-evaluation specifics) and treats it as controlling the strictness of the assessment: equal properties make any outcome difference a possible reproducibility failure, while differing properties make differences expected. On top of this, the framework pairs each of its four result types with scale-free measures of agreement or precision, so the same numbers can be read across studies.

What would settle it

Compile many pairs of studies that share all QRA++ experiment properties, including the same system outputs and metric implementation, and compare their CV* and P values with pairs that differ in properties; if matched pairs show no better reproducibility than unmatched ones, the claimed grounding of reproducibility in experiment similarity fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that reproducibility in NLP should be assessed quantitatively, as a degree, by treating evaluation as a measurement and reproducibility as measurement precision. Formally, for measurements $M_i$ that map a measurand $m$, object $O$, time $t_i$, and conditions $C_i$ to a quantity value $v_i$, reproducibility is $R(M_1,\dots,M_n) := \mathrm{Precision}(v_1,\dots,v_n)$, with repeatability as the special case where all $C_i$ are identical. QRA++ instantiates this by fixing a set of experiment properties as the conditions, by distinguishing four types of results (single scores, sets of scores, categorical labels, and findings about system differences), and by assigning each a family of degree-of-reproducibility measures: $\mathrm{CV}^{*}$ for single scores, Pearson and Spearman correlation plus Kendall's $\tau$ and $W$ for score sets, Fleiss's $\kappa$ and Krippendorff's $\alpha$ for labels, and $P$, the proportion of identical pairwise system ranks, for findings. Three example applications show higher degrees of reproducibility when experiment properties are the same, and clear differences by system type and evaluation method.

Load-bearing premise

The framework assumes that the chosen list of experiment properties is sufficient to determine when evaluation outcomes should match; the paper itself admits this list has no objective basis and serves as a way to set how strict the reproducibility assessment is.

Editorial extensions

If this is right

Reproducibility assessments from different studies become numerically comparable, because the chosen measures are unitless and defined uniformly.
A single reproduction attempt no longer decides success or failure; assessments over two or more comparable experiments yield a more stable degree of reproducibility.
Researchers can diagnose which experiment properties are associated with worse reproducibility (e.g., differing test datasets and seeds) and which system types are more fragile, as shown in the eight-experiment example.
The framework reveals that even when absolute scores reproduce only moderately, system rankings and findings can reproduce perfectly, so reproducibility conclusions depend on result type and level of aggregation.
Adding or removing properties from the experiment-property list changes the strictness of the assessment, giving researchers a dial for how demanding a reproducibility check should be.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step is to apply QRA++ across many existing reproduction studies to build a public corpus of quantitative reproducibility scores; the paper itself notes that such a body is currently lacking.
The same measurement-precision machinery could transfer to other fields that report evaluation metrics over shared tasks, such as information retrieval or vision-language evaluation, since the four result types are generic.
One testable extension is using QRA++ to monitor reproducibility over time within a fixed research community, checking whether improved reporting standards actually raise reproducibility scores.
Because the framework separates conditions (experiment properties) from outcomes, it could be combined with controlled perturbation studies that vary one property at a time, such as evaluator expertise or rating scale, to estimate its causal effect on reproducibility.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

QRA++ is a genuine attempt to give NLP a common yardstick for reproducibility, and the core framework is worth engaging with. The four result types, the three assessment levels, and the reporting template are a sensible synthesis; the small-sample-corrected CV* and the new P measure for pairwise rank agreement are reasonable choices. If you work on reproduction studies or shared tasks like ReproNLP, this gives you a concrete protocol to apply. Credit where due: the paper is clearly written, the metrology framing is used seriously, and the author explicitly flags the absence of an objective basis for the property set (Section 4). That honesty is a point in its favor.

The soft spots are mostly in the empirical demonstration, and one is load-bearing. The abstract claims 'clear evidence' that reproducibility depends on experiment similarity, but the examples are three convenience sets with no confidence intervals, no significance tests, and no raw score pairs provided. That alone would justify a conditional verdict. The sharper problem is Table 6. The paper splits the eight REPROLANG experiments into groups sharing 'same test dataset and random seed' versus different datasets and seeds, yet random seed is not among the Section 4 experiment properties, and the text in 7.3 says metric implementation and run-time environment also differ between subsets. So the comparison does not isolate the framework's own similarity variables. If seed is a variable that matters, the property set is incomplete; if it is not, Table 6 is not an application of QRA++. Either way, it cannot be the basis for the abstract's claim.

There are also small internal inconsistencies: Section 7.2 refers to 'three experiments' where there are two, and Table 4 is mislabelled as Table 3 in the text; Section 6.3 says Cohen's kappa for n=2 while Table 2 lists Fleiss's kappa. These are fixable but should be cleaned up.

Who should read this: anyone designing or adjudicating reproduction studies in NLP. The framework itself deserves referee time; the empirical claims need revision before it is published as is. I would send it to review, with a request that the author either make the Table 6 demonstration consistent with the property set or present it as a separate observation, and that the 'clear evidence' language be toned down to 'suggestive'. The framework is a net positive for the field.

Best, [You]

Referee Report

3 major / 5 minor

Summary. The paper presents QRA++, a framework for quantitative reproducibility assessment in NLP, built on metrology concepts. It defines a set of experiment properties (measurement conditions) in Table 1, four types of experimental results (Type I–IV), corresponding reproducibility measures (CV*, Pearson/Spearman/Kendall correlations, Fleiss's κ/Krippendorff's α, and a new P measure for ordinal findings), and three assessment levels (system, quality criterion, study). The framework is illustrated on three example sets of comparable experiments from ReproNLP 2024 and REPROLANG 2020. The paper's central claim, stated in the abstract, is that applying QRA++ reveals 'clear evidence' that degree of reproducibility depends on similarity of experiment properties, system type, and evaluation method.

Significance. The framework is a coherent synthesis of standard statistical measures with a metrological grounding, and the explicit specification of experiment properties is a useful step toward comparable reproducibility assessments. The paper provides careful definitions, including the small-sample-corrected CV* and the P measure. However, the empirical support for the central claim is currently weak: the key comparison in Table 6 is confounded and uses a variable (random seed) not in the property set, the property set is admitted to lack an objective basis, and no inferential statistics are provided. If the property set can be validated and the examples restricted to the framework's own variables, the approach would be a valuable contribution; as it stands, the abstract overstates the evidence.

major comments (3)

[§7.3, Table 6] The comparison in Table 6 groups experiments by 'same test dataset and random seed' versus 'different datasets and seeds,' but random seed is not among the experiment properties in Table 1 nor in the HEDS 3.0 questions in Appendix A. Therefore, Table 6 does not instantiate the QRA++ property-based expectation mechanism: either the property set is incomplete (in which case the grounding in Section 4 is not minimal), or the comparison is made on a variable outside the framework (in which case it cannot support the abstract's claim of 'clear evidence' about property similarity). Furthermore, Section 7.3 states that metric implementation and run-time environment also differ between the two subsets, so the 'same dataset and seed' subset is not homogeneous with respect to the Section 4 properties; the observed differences in Table 6 are confounded. No confidence intervals or significance tests are reported; with only four experiments per subset, differences such as P=0.706 versus P=0.558 are within plausible sampling variability.
[§4] The paper explicitly concedes that the minimal set of experiment properties is 'not a distinction for which there is an objective basis' and calls it 'a way of controlling the strictness of the reproducibility assessment.' This is a load-bearing limitation for the framework's central claim (abstract item iii) that QRA++ 'grounds expectations about degree of reproducibility in degree of similarity between experiments.' If the property set is arbitrary, those expectations are not externally valid, and the framework can only describe outcome differences, not explain them. The authors should either validate the property set on a larger body of reproduction studies or substantially weaken this claim in the abstract and conclusion.
[Abstract and §9] The abstract and conclusion claim that applying QRA++ reveals 'clear evidence' about factors affecting reproducibility. The evidence base is limited to three illustrative example sets, with no hypothesis tests, confidence intervals, or correction for multiple comparisons. For instance, Table 6 compares four experiments per group; with n=4, the differences in mean CV* (8.7 vs 14.9) and P (0.706 vs 0.558) are not shown to be statistically reliable. The paper should reframe these as illustrative findings rather than 'clear evidence,' or provide appropriate statistical support.

minor comments (5)

[§7.2] The text says 'Table 3 reports QRA++ results based on two comparable experiments,' but the table referred to is Table 4; it also says 'the experiment properties ... were exactly the same in all three experiments' when there are only two experiments.
[§6.3] There is a typo, 'beccause,' and the text says 'Cohen's κ for n = 2, and Krippendorff's α for n>2,' but Table 2 appears to list both κ and α for both cases; please clarify which measure applies when.
[Table 2] The note under the table says 'P = proportion of differences between pairs of systems that have the same sign (see Section 1),' but the measure is defined in Section 6.4, not Section 1.
[§6.1] The thresholds for 'good reproducibility' (below about 12 for human evaluations, below 1 for metric-based evaluations) are stated without a citation or derivation; please provide a source or describe how these values were determined.
[Figure 1] The formula for P uses the sign function; please clarify how zero differences are treated (e.g., if one experiment yields exactly equal scores for a pair of systems and another does not).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: QRA++'s reproducibility measures are standard statistics applied to externally published score pairs, and the property-based expectation is an explicit, untested assumption rather than a fitted or definitionally derived result.

full rationale

QRA++ does not fit parameters, and no prediction in the paper is equivalent to its inputs by construction. The reproducibility measures (CV*, r, rho, tau, W, kappa, alpha, P) are standard statistics computed from score pairs taken from independently published experiments: August et al. 2022 with its ReproNLP replications, Gabriel et al. 2022 with Mahlaza et al. 2024, and the REPROLANG 2020 replications. Equations (2) and (3) are standard small-sample coefficient-of-variation formulas, and Equation (4) is an explicit pairwise sign-agreement proportion; neither is calibrated to force a desired conclusion. The property-based expectation in Section 4 is openly presented as a chosen assumption, with the paper conceding 'this is not a distinction for which there is an objective basis,' so the claim that reproducibility depends on experiment similarity is an interpretive claim about the worked examples, not a tautology. The self-citations (Belz 2022 for CV* and the metrological framing; Belz and Thomson 2024b for HEDS 3.0, used as the property vocabulary) are conventions and reporting instruments rather than load-bearing theorems that presuppose the target results. The closest thing to an empirical test, Table 6, is statistically weak: n=4 per subset, no significance testing, and random seed is used as a grouping criterion even though it is not among the Table 1 properties; those are correctness and evidential limitations, not circular reductions. No equation in the paper makes the assessment equal to its inputs by definition, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The framework adds no fitted numeric parameters beyond interpretive thresholds. It rests on the mapping from metrology precision to reproducibility, on the sufficiency of the selected experiment properties for comparability (explicitly acknowledged as lacking objective basis in Section 4), and on the alignment of the example data sets. The P measure and reporting template are new, while CV*, correlation, and agreement measures are standard statistics imported from the literature.

free parameters (1)

CV* good-reproducibility thresholds = below 12 (human eval), below 1 (metric eval)
Section 6.1 sets interpretive thresholds for 'good' system-level reproducibility based on reports from the author's own ReproNLP shared task series. These are hand-selected reference values, not fitted or derived in this paper, and they affect the qualitative interpretation of the example results.

assumptions (5)

domain assumption Reproducibility is measurement precision under specified conditions, following VIM definitions adopted from Belz (2022).
Section 2 identifies reproducibility with precision of measured quantity values; the entire QRA++ measure set follows from this mapping.
domain assumption The selected experiment properties are the minimal set such that identical values imply expected identical outcomes; differing values imply expected outcome differences.
Section 4 states this distinction has no objective basis and is a way to control strictness. This is load-bearing for using experiment similarity to set reproducibility expectations.
domain assumption The example studies constitute aligned comparable experiments with the claimed property matches.
Sections 7.1-7.3 assume the cited ReproNLP and REPROLANG studies can be treated as comparable experiments with the same systems and quality criteria; raw score pairs and explicit alignment checks are not provided.
standard math Normal-theory approximations hold for the unbiased sample standard deviation s* and its standard error.
Section 6.1 uses the c4(n) correction and t-distribution confidence intervals from Rao (1973), which require normality or large-sample approximation.
domain assumption Each experiment's labels can be treated as one rater for inter-rater agreement measures.
Section 6.3 reinterprets kappa and Krippendorff's alpha as reproducibility measures by treating all labels from an experiment as produced by one rater; this is plausible but not empirically validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing." pith.science (2026). https://pith.science/paper/6ENAHGX6

@misc{pith2026250517043,
  author       = {Pith},
  title        = {Pith review of: QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ENAHGX6}},
  note         = {Machine review of arXiv:2505.17043}
}

read the original abstract

Reproduction studies reported in NLP provide individual data points which in combination indicate worryingly low levels of reproducibility in the field. Because each reproduction study reports quantitative conclusions based on its own, often not explicitly stated, criteria for reproduction success/failure, the conclusions drawn are hard to interpret, compare, and learn from. In this paper, we present QRA++, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilises reproducibility measures that are directly comparable across different studies; and (iii) grounds expectations about degree of reproducibility in degree of similarity between experiments. QRA++ enables more informative reproducibility assessments to be conducted, and conclusions to be drawn about what causes reproducibility to be better/poorer. We illustrate this by applying QRA++ to three example sets of comparable experiments, revealing clear evidence that degree of reproducibility depends on similarity of experiment properties, but also system type and evaluation method.

Figures

Figures reproduced from arXiv: 2505.17043 by the authors.

**Figure 1.** P, the proportion of identical pairwise system ranks in a set of comparable experiments E. propose to use standard inter-rater agreement measures, namely Cohen’s κ for the case of n = 2, and Krippendorff’s α for n > 2. In other contexts these measures typically assess (literally) the agreement between two raters and more than two raters, respectively. When using them to assess reproducibility, we consider all lab… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

AAH/USFWS. n.d. Assay validation methods: D efinitions and terms. Aquatic Animal Health Program, U.S. Fish & Wildlife Service

work page
[4]

SE Ahmed. 1995. A pooling methodology for coefficient of variation. Sankhy \=a : The Indian Journal of Statistics, Series B , pages 57--75

work page 1995
[5]

Cristina Arhiliuc, Jelena Mitrovi \'c , and Michael Granitzer. 2020. https://www.aclweb.org/anthology/2020.lrec-1.690 Language proficiency scoring . In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5624--5630, Marseille, France. European Language Resources Association

work page 2020
[6]

Tal August, Katharina Reinecke, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.acl-long.569 Generating scientific definitions with controllable complexity . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8298--8317, Dublin, Ireland. Association for Computational Linguistics

doi:10.18653/v1/2022.acl-long.569 2022
[7]

Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. https://aclanthology.org/W13-2322 A bstract M eaning R epresentation for sembanking . In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178--186...

2013
[8]

Anya Belz. 2022. A metrological perspective on reproducibility in NLP . Computational Linguistics, 48(4):1125--1135

work page 2022

Show all 29 references

[9]

Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. https://aclanthology.org/2021.eacl-main.29 A systematic review of reproducibility research in natural language processing . In Proceedings of the 16th Conference of the European Chapter of the Association ...

2021
[10]

Anya Belz, Simon Mille, and Craig Thomson. 2025. A taxonomy of quality criterion names and definitions for evaluating nlp systems in terms of standard comparable aspects of quality

2025
[11]

Anya Belz, Simon Mille, Craig Thomson, and Rudali Huidrom. 2024. https://aclanthology.org/2024.inlg-demos.4 QCET : An interactive taxonomy of quality criteria for comparable and repeatable evaluation of NLP systems . In Proceedings of the 17th International Natural Language Ge...

2024
[12]

Anya Belz and Craig Thomson. 2023. https://aclanthology.org/2023.humeval-1.4 The 2023 R epro NLP shared task on reproducibility of evaluations in NLP : Overview and results . In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, pages 35--48, Varna, Bulgaria. ...

2023
[13]

Anya Belz and Craig Thomson. 2024 a . The 2024 repronlp shared task on reproducibility of evaluations in nlp: Overview and results. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval)@ LREC-COLING 2024, pages 91--105

2024
[14]

Anya Belz and Craig Thomson. 2024 b . Heds 3.0: The human evaluation data sheet version 3.0. arXiv preprint arXiv:2412.07940

2024 arXiv
[15]

Yves Bestgen. 2020. https://www.aclweb.org/anthology/2020.lrec-1.687 Reproducing monolingual, multilingual and cross-lingual CEFR predictions . In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5595--5602, Marseille, France. European Language Resou...

2020
[16]

Ant \'o nio Branco, Nicoletta Calzolari, Piek Vossen, Gertjan Van Noord, Dieter van Uytvanck, Jo \ a o Silva, Lu \' s Gomes, Andr \'e Moreira, and Willem Elbers. 2020. https://www.aclweb.org/anthology/2020.lrec-1.680 A shared task of a new, collaborative type to foster reprodu...

2020
[17]

Andrew Caines and Paula Buttery. 2020. https://www.aclweb.org/anthology/2020.lrec-1.689 REPROLANG 2020: Automatic proficiency scoring of C zech, E nglish, G erman, I talian, and S panish learner essays . In Proceedings of The 12th Language Resources and Evaluation Conference, ...

2020
[18]

Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.acl-long.222 Misinfo reaction frames: Reasoning about readers ' reactions to news headlines . In Proceedings of the 60th Annual Mee...

2022 doi
[19]

Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A

David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. https://doi.org/10.18653/v1/2020.inlg-1.23 Twenty years of confusion in human evaluation: NLG ne...

2020 doi
[20]

Eva Huber and C a g r C \"o ltekin. 2020. https://www.aclweb.org/anthology/2020.lrec-1.688 Reproduction and replication: A case study with automatic essay scoring . In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5603--5613, Marseille, France. Eu...

2020
[21]

JCGM. 2012. https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf International vocabulary of metrology: B asic and general concepts and associated terms ( VIM ) . Joint Committee for Guides in Metrology. https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf

2012
[22]

Kamp and U

H. Kamp and U. Reyle. 2013. From Discourse to Logic: Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory. Kluwer, Dordrecht

2013
[23]

Yiru Li, Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2024. https://aclanthology.org/TODO.27 Report for reprohum project 0033 . In Proceedings of the 4th Workshop on Human Evaluation of NLP Systems, Turin, Italy. Association for Computational Linguistics

2024
[24]

Maria Keet

Zola Mahlaza, Toky Raboanary, Kyle Seakgwa, and C. Maria Keet. 2024. https://aclanthology.org/TODO.34 Another evaluation of readers' reactions to news headlines . In Proceedings of the 4th Workshop on Human Evaluation of NLP Systems, Turin, Italy. Association for Computational...

2024
[25]

Maja Popovi \'c and Anya Belz. 2022. On reporting scores and agreement for error annotation tasks. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 306--315

2022
[26]

Calyampudi Radhakrishna Rao. 1973. Linear statistical inference and its applications. Wiley

1973
[27]

Sokal and F.J

R.R. Sokal and F.J. Rohlf. 1971. Biometry: The Principles and Practice of Statistics in Biological Research. WH Freeman

1971
[28]

Sowmya Vajjala and Taraka Rama. 2018. https://doi.org/10.18653/v1/W18-0515 Experiments with universal CEFR classification . In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications , pages 147--153, New Orleans, Louisiana. Associ...

2018 doi
[29]

Emiel van Miltenburg, Anouck Braggaar, Nadine Braun, Martijn Goudbeek, Emiel Krahmer, Chris van der Lee, Steffen Pauws, and Frédéric Tomas. 2024. https://aclanthology.org/TODO.9 How reproducible are fluency ratings of generated text? a reproduction of august et al. 2022 . In P...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

AAH/USFWS. n.d. Assay validation methods: D efinitions and terms. Aquatic Animal Health Program, U.S. Fish & Wildlife Service

work page

[4] [4]

SE Ahmed. 1995. A pooling methodology for coefficient of variation. Sankhy \=a : The Indian Journal of Statistics, Series B , pages 57--75

work page 1995

[5] [5]

Cristina Arhiliuc, Jelena Mitrovi \'c , and Michael Granitzer. 2020. https://www.aclweb.org/anthology/2020.lrec-1.690 Language proficiency scoring . In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5624--5630, Marseille, France. European Language Resources Association

work page 2020

[6] [6]

Tal August, Katharina Reinecke, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.acl-long.569 Generating scientific definitions with controllable complexity . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8298--8317, Dublin, Ireland. Association for Computational Linguistics

doi:10.18653/v1/2022.acl-long.569 2022

[7] [7]

Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. https://aclanthology.org/W13-2322 A bstract M eaning R epresentation for sembanking . In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178--186...

2013

[8] [8]

Anya Belz. 2022. A metrological perspective on reproducibility in NLP . Computational Linguistics, 48(4):1125--1135

work page 2022