Computation and Language · arXiv:2610.35423 v1 · submitted 26 Oct 2026 · 36 references
INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM
The paper says
“Operator manuals of agricultural implements are delivered in many languages, and the standards require that competent persons review every translation, which makes this review the bottleneck.”
more of the abstract
This paper presents a proof of concept that compares two training workflows on the same LLM, EuroLLM-1.7B-Instruct, trained on operator manuals of one manufacturer. In approach A, a back-translator translates each language version back into English, and an English reader compares it with the English version. In approach B, a checker reads the English version and the translation together, where the checker reports differences. The test sets contain errors planted in numbers, units, opposite words, obligations, negations, and conditions in 24 languages, and they include unmodified text and meaning-preserving edits for alerts and false alarms. The trained back-translator concealed many planted errors, most of all weakened obligations, while translation-quality scores rose. In contrast, the separately trained checker detected most planted errors, although it was weak for units and for number changes unlike those in training. A check by the author confirmed the concealment. In addition, a simple rule check caught the number and unit changes, but it alerted on harmless rewrites. A check of alerts on real Finnish translations showed that most alerts were false, caused by language-specific wording, so the precision on real text remains a limitation. Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful head-to-head showing that fine-tuning a back-translator hides the errors you care about while a fine-tuned checker catches them — within a controlled planted-error setting whose transfer to real translations the paper itself shows is shaky. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 36 references say.
The paper says, at §4.2 and §5.4
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
Section 4.2 states that using different planters for training and test 'measures whether the checker generalises to kinds of changes it has not seen.' This is not supported for the four word-level types. The training and test plantings are produced by different models (gpt-oss-120b and Gemma 4), but both pass the same acceptance filter: exactly one changed span, numbers kept, …
more of objection 1
… a length ratio of 0.7 to 1.3 (0.4 to 1.0 for conditions), and Clef acceptance at probability at least 0.5 (Table 3; Sections 3.3 and 4.2, where the word-level training plantings are said to be accepted 'using the same automatic checks and Clef threshold as the test sets'). A different generator behind the same filter is a different instance of the same error class, not a different kind of change. The paper's own data support this reading: for numbers and units, where the training and test changes genuinely differ in structure (training doubles, halves, or multiplies by 1.5 and swaps mm/m, kg/t, bar/MPa, km/h/km; test multiplies or divides by ten, changes one digit, and swaps Nm/N, cm/mm, kW/W, mm/cm), detection transfers poorly (62% for units, 4% to 55% for division by ten, 0% for Nm to N), whereas the word-level types, which are structurally identical up to the planter, reach 93% to 98%. The sentence in Section 5.4 that the checker 'learned the kinds of changes it saw in training' is in tension with the Section 4.2 generalization claim. I request rewording the Section 4.2 claim and adding an explicit discussion of the shared error notion between checker training and test, so that the headline 90% figure is presented as within-class detection of one-span, Clef-validated changes rather than as evidence of generalization to new kinds of discrepancies.
The paper says, at §5.4 and §6.6
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The Finnish real-text check bears on recall as well as precision, and the carry-over caveat should be extended accordingly. The paper reports that 19 of 20 checker alerts on real Finnish units were false and, separately, that 2 of the 20 unflagged units actually differed (one added word, one alignment error).
more of objection 2
Section 5.4 concludes that 'the detection rates of planted errors do not carry over to the precision of alerts on real translations'; the abstract and conclusion state that precision on real text is a limitation. This is acknowledged and appreciated, but the unflagged differences are evidence that the carry-over problem concerns detection (recall) as well: genuinely different natural pairs sit inside the 'Same meaning.' class. The mechanism the paper partly describes — the correct-pair filter of Section 4.2 removed exactly the legitimate forms ('tulee', positive or restrictive negation constructions) that the untuned back-translator flagged, so the checker never learned them as correct — explains both directions, and Section 6.2 already gestures at it. I ask the authors to (i) state in Section 5.4 that the Finnish check limits claims about detection rates on real translator errors in both directions (false alerts and missed differences), not only precision; and (ii) say explicitly that no result links the planted-setting detection rates to translator-produced errors, given that the test sets contain only planted errors (an acknowledged limitation in Section 6.6). A concrete and feasible addition within the paper's scope would be to report the checker's verdicts on a sample of the 2,134 corpus pairs flagged during alignment (Section 3.1), which plausibly contain natural discrepancies.
6 minor objections
The paper says, at §4.4 and §5.2
Pith says: minor objection 1 of 6
The disclosure that the comparison of the two workflows was 'framed after the first results of the checker' (Section 4.4) is honest, but it should be repeated where the opposite-direction result is presented, with an explicit label of that comparison as exploratory; the pre-registered hypothesis H1 concerns hiding in close versus new text and supports only the approach A half of the headline.
The paper says, at Table 5
Pith says: minor objection 2 of 6
Add a footnote to Table 5 stating that the approach A condition row is not interpretable per Section 5.3 (the element rules agreed with the author on only 42% of the 19 condition items and Clef on 32%), since the pooled rows ('all planted errors') include conditions.
show 4 more
The paper says, at §4.5 and §5.3
Pith says: minor objection 3 of 6
The concealment check rests on a single judge who is also the author. The 9 of 9 versus 0 of 9 result on weakened obligations is striking but small; a second annotator on at least the obligation items would substantially strengthen the claim that the concealment is real for human readers.
The paper says, at §4.1 and §4.2
Pith says: minor objection 4 of 6
The printed prompt transcripts read 'Y ou' instead of 'You' in the system message of both prompts; please fix this typo or confirm that it is an artifact of the transcript rather than the actual prompt.
The paper says, at Reproducibility
Pith says: minor objection 5 of 6
The paper reports model revisions and hyperparameters in Table 3, but it does not state whether INVAR-Agri, the planting records, and the fine-tuning code will be released. Since the paper constructs new test sets, a public repository or a clear data-availability statement would materially support reproduction.
The paper says, at §3.3
Pith says: minor objection 6 of 6
The Claude audit of the 16 added languages found 7 of 192 (3.6%) invalid plantings, concentrated in obligation, negation, and condition. As a robustness check, please report the per-type detection rates with these items excluded, or state why exclusion would not affect the conclusions.
What would settle it
Evaluate the fine-tuned checker on naturally occurring discrepancies: have a bilingual reviewer independent of the author mark genuine differences of the six error types between the English version and two or three other language versions of the two test manuals, restricted to complete sentences and paragraphs, without any Clef validation or one-span filtering; then run the trained checker at its fixed prompt and alert operating point and compare detection and alert precision against the planted rates. If detection on natural errors falls materially below roughly 90% or precision stays near the roughly 5% level seen in the Finnish check, the practical transfer claim is not supported by the current evidence; if detection and precision stay high, the concern is defused.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The headline result holds up in the controlled setting: fine-tuning a back-translator on the manufacturer's own manuals hides the exact planted errors a reviewer looks for (weakened obligations drop from 94% to 30% detected), while a checker fine-tuned on the same LLM catches most of them (22% to 90% without conditions). That opposite direction is the genuinely new thing here. Round-trip repair and memorisation were known separately; the head-to-head comparison on the same LLM, same manuals, same test items does not appear in the cited literature.
The paper earns its credibility. The design has three seeds, bootstrap intervals over whole texts and pages, a pre-registered interaction hypothesis with recorded amendments, a blind author check of 117 back-translations (9/9 vs 0/9 on weakened obligations), and multi-model validation of the plantings. The checker's word-level generalisation from gpt-oss-120b-planted training errors to Gemma-4-planted test errors is genuine evidence against pure surface matching. The author also states the limitations directly, including the Finnish real-text check showing that most alerts were false.
The soft spots are real, though they do not break the controlled comparison. First, no code, data, or models are released, so the numbers are not independently reproducible. Second, transfer from planted to real discrepancies is unestablished, and the stress-test concern has teeth: every planted test item passes the same one-span, length-ratio, and Clef-probability filters used to build the checker's training data (Sections 3.3, 4.2), so the checker is rewarded for flagging exactly the class of changes those filters define as errors. The Finnish check shows what naturally written pairs look like: 19 of 20 alerts were false, triggered by normal wording like 'tulee', and 2 of 40 unflagged units differed. The paper hedges the practical claim to 'might support', which is the right amount of confidence, but the abstract's 90% figure will be over-read. Third, the test manuals overlap training text heavily (69% and 42% word-for-word), though the checker's detection is flat across similarity groups, so this matters less for the main result. Single-author human judging and small cells in some language/type combinations are minor issues. The citation pattern looks sound; the ingredients are credited and the novelty claim is properly scoped.
This is a careful proof of concept, not a benchmark. It deserves a serious referee: the controlled result is a useful cautionary tale for anyone fine-tuning MT on safety-critical documentation, and the transfer gap is the obvious next step. Send it to review, with a request for artifacts.
The strongest objection
The opposite-direction result within the controlled setting is well supported: three seeds, 24 languages, a pre-registered interaction test with all intervals above zero, and an author blind check (9/9 vs 0/9 on weakened obligations). The load-bearing gap is the transfer from planted to real discrepancies, and there is a specific mechanism the paper does not fully spell out. Every planted test item shares a structure with the checker's own training plantings: exactly one changed span, accepted by the same automatic one-span/length rules and the same Clef validator at a probability threshold of at least 0.5 (Sections 3.3 and 4.2). The checker is therefore rewarded for flagging precisely the class of changes that Clef and these rules deem valid errors, and training and test share that notion of 'error'. Real translation behaviour differs: in the paper's Finnish check, translators restructure sentences and use legitimate surface forms such as 'tulee' for 'must', forms the checker's training filter (which removed all correct pairs where the untuned back-translation disagreed on number, unit, obligation or negation, Section 4.2) never taught it to accept. The consequence is visible in the paper's own data: 19 of 20 alerts on real Finnish units were false, and of 20 unflagged Finnish units two actually differed (Section 5.4). The cross-planter generalization for word-level types (gpt-oss-120b to Gemma 4, 93-98%) is genuine evidence against pure surface matching, but both planters' outputs pass identical Clef and one-span filters, so it does not escape the shared-filter structure. The paper is honest about this, hedging the practical claim to 'might support', and the controlled comparison itself is not broken; what is at stake is the interpretation a reader takes from the headline 90% detection figure as a guide to real review support.
I read the paper in good faith: it is an unusually careful, honestly scoped empirical comparison with a pre-registered hypothesis, three seeds, bootstrap intervals, explicit protocol amendments, and a dramatic author blind check (9/9 untuned vs 0/9 fine-tuned on weakened obligations). The opposite-direction finding between approach A and approach B on identical test items is internally consistent and survives the noise checks the paper itself reports: counting garbled back-translations as visible changes the approach A fall only from 87%-to-67% to 95%-to-77%; a lenient reading of the untuned checker's off-format answers changes the approach B gain only from +68 to +66; the pre-registered interaction terms have all intervals above zero, including under the sensitivity bootstrap. I give credit for the cross-planter generalization on word-level types, which is a real generalization test, and for the explicit limitation statements. The most load-bearing concern is exactly the one the reader identified, and I find no internal inconsistency in the controlled comparison, so no reason to move the verdict. My addition is to sharpen the mechanism: the test plantings and the checker's training plantings are filtered by identical one-span/length rules and the identical Clef validator, so training and test share a structural notion of what an error is; the Finnish real-text result (19/20 flagged alerts false, and a genuine difference among the 20 unflagged units) is the paper's own evidence that this shared notion does not match naturally occurring translation behaviour. The paper acknowledges this and hedges to 'might support', which is why CONDITIONAL, not REJECT, remains the right verdict: the practical claim is unestablished but not the controlled comparison. The concern does not change the verdict, so I agree with the reader and recommend UNCHANGED.
The referee's report
If the controlled results hold, the paper makes a valuable and clearly scoped demonstration. The opposite-direction effect on identical test items is supported by three seeds, bootstrap intervals over whole English texts and pages, a pre-registered interaction hypothesis with recorded protocol amendments, multi-model validation of plantings (Clef, Claude), and a blind author check that agrees with the element rules on 85% of 117 items. The design deserves credit for planting checker training errors with a different model (gpt-oss-120b) than the test errors (Gemma 4), for full-parameter fine-tuning, and for reporting seed ranges and interval-based uncertainty instead of point estimates. The manuscript is unusually honest: it reports the 19/20 false-alert rate on real Finnish text, the 2 of 40 unflagged differing units, the 0% detection of Nm to N, and the absence of translator-made errors in the test sets, and it does not claim system readiness.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tests two ways a single English-reading expert could check operator manuals that ship in up to 24 languages: back-translate each version and compare it with the English original (approach A), or have the model read the English and the translation together and report differences (approach B). On the same LLM, the same manuals, and the same test items, fine-tuning moved the two workflows in opposite directions: the trained back-translator concealed planted errors, with detection of weakened obligations falling from 94% to 30% even as translation-quality scores rose, while the trained checker's detection of planted errors rose from 22% to 90%. The point of the paper is that a metric like COMET or chrF can certify a back-translator that is quietly getting less faithful, so a fine-tuned back-translator is the wrong support tool for the review the standards require. The trained checker is the more promising direction, but the paper's own Finnish check shows most of its alerts on real translations are false, so it can support, not replace, the review by competent persons, and firmer conclusions need more data.
What carries the argument. The load-bearing objects are the two fine-tuning workflows on EuroLLM-1.7B-Instruct, a back-translator (approach A) and a direct checker (approach B), evaluated on INVAR-Agri, test sets of 5332 planted errors in six types (number, unit, polarity, obligation, negation, condition) across 24 languages, plus 2016 unmodified units and 432 meaning-preserving edits. The mechanism that carries the argument is the training target: approach A is rewarded, via chrF on development data, for reproducing the English side of multi-parallel pairs, which teaches conventions such as "English instructions carry 'must'" and memorised text, so it repairs planted errors on the way back; approach B is rewarded, via balanced accuracy, for answering "Check:" when a difference is present, so it reports differences rather than restoring them. Detection is decided by element rules that search back-translations for the planted English value, classifying outcomes as carried, hidden, or garbled, and by the verdict form of the checker, with the author's blind review of 117 back-translations as the human check on the concealment.
Core claim. The paper's central result is that fine-tuning the same 1.7-billion-parameter multilingual model on one manufacturer's operator manuals pushed two translation-checking workflows in opposite directions on identical test items. With the back-translator of approach A, detection of planted errors without conditions fell from 87% to 67% (from 81% to 65% with conditions), and the largest collapse was for weakened obligations, "must" quietly turned into "should", which dropped from 94% untuned to 30% fine-tuned, while the translation-quality scores COMET and chrF rose. The author's blind check confirmed that the concealment is real for a human reader: the planted error was visible in 9 of 9 untuned back-translations but in 0 of 9 fine-tuned ones. With the directly trained checker of approach B, detection without conditions rose from 22% to 90%, with the four word-level error types at 93% to 98%, although units and number changes unlike those in training transferred only partly. The paper interprets approach A's behaviour as the back-translator learning, from roughly 20 training pairs per English sentence, that English instructions say "must" whatever the other language says, so it restores the expected instruction on the way back; approach B instead learns the difference itself as its target.
The premise it rests on. The evaluation rests on the assumption that planted synthetic errors, mostly in text that overlaps the training manuals, are a fair stand-in for the mistakes real translators make: the test sets contain no translator-made errors, and the paper's own Finnish check showed 19 of 20 checker alerts on real translations were false.
If the paper is right (Pith's inferences, not the authors')
A manufacturer who fine-tunes a back-translator on its own manuals to raise translation quality can make back-translation review less safe: detection of weakened obligations fell from 94% to 30% while COMET and chrF rose, so standard quality metrics would not reveal the loss.
A checker trained on synthetic errors from the same manuals detects 93% to 98% of word-level planted errors and, combined with a deterministic number and unit rule check, 97% of errors without conditions at 4% alerts, which is a viable division of labour for supporting review.
The checker's transfer is partial: it beats the back-translator on numbers (26% to 85%) and units (15% to 62%), yet it never detected "Nm" written as "N", so its training data determines which discrepancies it can see.
An alert rate of about 3% on unmodified text is not a measured reduction of review work: most alerts on real Finnish translations (19 of 20) were false, triggered by normal wording such as the obligation form "tulee".
The hiding effect grows with training intensity and is strongest in text close to the training manuals, but it also appears in new text, so the risk is not limited to verbatim copying.
The same mechanism should be probed in other repetitive document domains, such as medical instructions, legal notices, or safety signage, where a fine-tuned back-translator's concealment would carry the same hazard; a cheap audit is the paper's blind protocol, having a reviewer decide which of two English texts a back-translation matches.
Because the checker's false alerts in Finnish came from just two surface cues, "tulee" read as "should" and negation reported missing where Finnish used a restrictive form, a concrete prediction is that adding those forms as "Same meaning." training examples would raise Finnish alert precision without hurting planted-error detection.
The test sets contain only planted errors and no errors made by translators, so a natural extension is to run the same two workflows on errors extracted from real translation revisions, where changes are subtler and wording less contrived, and see whether the 90% detection and the 30% concealment survive.
If the training target is the cause of the opposite directions, then training the back-translator on pairs that keep an error, the checker's filtered data, should reduce concealment; this is a testable variant of a next step the paper itself names.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Operator manuals of agricultural implements are delivered in many languages, and the standards require that competent persons review every translation, which makes this review the bottleneck. This paper presents a proof of concept that compares two training workflows on the same LLM, EuroLLM-1.7B-Instruct, trained on operator manuals of one manufacturer. In approach A, a back-translator translates each language version back into English, and an English reader compares it with the English version. In approach B, a checker reads the English version and the translation together, where the checker reports differences. The test sets contain errors planted in numbers, units, opposite words, obligations, negations, and conditions in 24 languages, and they include unmodified text and meaning-preserving edits for alerts and false alarms. The trained back-translator concealed many planted errors, most of all weakened obligations, while translation-quality scores rose. In contrast, the separately trained checker detected most planted errors, although it was weak for units and for number changes unlike those in training. A check by the author confirmed the concealment. In addition, a simple rule check caught the number and unit changes, but it alerted on harmless rewrites. A check of alerts on real Finnish translations showed that most alerts were false, caused by language-specific wording, so the precision on real text remains a limitation. Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 0.99
“Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data.”
How the main claim is stated: speculative 0.73
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.98 · human participants 0.13 · controlled comparison 0.65 · randomized assignment 0.02 · states quantitative result 0.23 · compares with prior results 0.90 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Operator manuals of agricultural implements are delivered in many languages, and the standards require that competent persons review every translation, which makes this review the bottleneck. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| This paper presents a proof of concept that compares two training workflows on the same LLM, EuroLLM-1.7B-Instruct, trained on operator manuals of one manufacturer. | contribution 0.98 | 0.83 | 0.04 | asserted 0.96 |
| In approach A, a back-translator translates each language version back into English, and an English reader compares it with the English version. | method 0.99 | 0.71 | 0.02 | asserted 0.97 |
| In approach B, a checker reads the English version and the translation together, where the checker reports differences. | method 0.99 | 0.65 | 0.02 | asserted 0.92 |
| The test sets contain errors planted in numbers, units, opposite words, obligations, negations, and conditions in 24 languages, and they include unmodified text and meaning-preserving edits for alerts and false alarms. | method 0.99 | 0.85 | 0.03 | asserted 1.00 |
| The trained back-translator concealed many planted errors, most of all weakened obligations, while translation-quality scores rose. | result 1.00 | 0.92 | 0.12 | asserted 1.00 |
| In contrast, the separately trained checker detected most planted errors, although it was weak for units and for number changes unlike those in training. | result 1.00 | 0.94 | 0.25 | asserted 1.00 |
| A check by the author confirmed the concealment. | result 0.75 | 0.90 | 0.04 | asserted 1.00 |
| In addition, a simple rule check caught the number and unit changes, but it alerted on harmless rewrites. | result 0.94 | 0.85 | 0.03 | asserted 1.00 |
| A check of alerts on real Finnish translations showed that most alerts were false, caused by language-specific wording, so the precision on real text remains a limitation. | limitation 0.95 | 0.94 | 0.08 | asserted 0.99 |
| Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data. | interpretation 0.62 | 0.55 | 0.78 | speculative 0.73 |
What kind of paper the classifier reports this is: empirical study 0.78 · evidence from observation or experiment 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 8,186 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
References (36)
- [1] T. Oksanen and J. Gun, “Machine-readable operator manuals over the ISO 11783 network for offline, multilingual LLM assistance,”Computers and Electronics in Agriculture, 2026, accepted for publication; preprint arXiv:2610.31863. [Online]. Available: https://arxiv.org/abs/2610.31863 on PithPith's reviewarXiv
- [2] N. Mehandru, S. Agrawal, Y . Xiao, G. Gao, E. C. Khoong, M. Carpuat, and N. Salehi, “Physician detection of clinical harm in machine trans- lation: Quality estimation aids in reliance and backtranslation identifies critical errors,” inProceedings of the 2023 Conference on Empi... on Pith
- [3] H. Somers, “Round-trip translation: What is it good for?” in Proceedings of the Australasian Language Technology Workshop 2005, Sydney, Australia, Dec. 2005, pp. 127–133. [Online]. Available: https://aclanthology.org/U05-1019/ on Pith
- [4] L. Specia, F . Blain, M. Fomicheva, C. Zerva, Z. Li, V. Chaudhary, and A. F . T. Martins, “Findings of the WMT 2021 shared task on quality estimation,” inProceedings of the Sixth Conference on Machine Translation (WMT). Online: Association for Computational Linguistics, Nov. 2... on Pith
- [5] N. M. Guerreiro, R. Rei, D. van Stigt, L. Coheur, P . Colombo, and A. F . T. Martins, “xCOMET: Transparent machine translation evaluation through fine-grained error detection,”Transactions of the Association for Computational Linguistics, vol. 12, pp. 979–995, 2024. [6]IEC/IEE... on Pith
- [7] P . H. Martins, P . Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P . Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F . T. Martins, “EuroLLM: Multilingual language models for Europe,” arXiv preprint arX... on PithPith's reviewarXiv
- [8] J. Moon, H. Cho, and E. L. Park, “Revisiting round-trip translation for quality estimation,” inProceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT). Lisboa, Portugal: European Association for Machine Translation, Nov. 2020, pp. 9... on Pith
- [9] S. Agrawal, N. Mehandru, N. Salehi, and M. Carpuat, “Quality esti- mation via backtranslation at the WMT 2022 quality estimation task,” inProceedings of the Seventh Conference on Machine Translation (WMT). Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational... on Pith
- [10] V. Zouhar, M. Novák, M. Žilinec, O. Bojar, M. Obregón, R. L. Hill, F . Blain, M. Fomicheva, L. Specia, and L. Y ankovskaya, “Backtrans- lation feedback improves user confidence in MT, not quality,” inPro- ceedings of the 2021 Conference of the North American Chapter of the Ass... on Pith
- [11] D. Ki, K. Duh, and M. Carpuat, “Should I share this translation? Evaluating quality feedback for user reliance on machine translation,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Suzhou, China: Association for Computation... on Pith
- [12] M. J. Martindale and M. Carpuat, “Fluency over adequacy: A pilot study in measuring user trust in imperfect MT,” inProceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track). Boston, MA, USA: Association for Machine... on Pith
- [13] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, “COMET: A neural framework for MT evaluation,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). On- line: Association for Computational Linguistics, Nov. 2020, pp. 2685– 2702 on Pith
- [14] Y .-L. Tuan, A. El-Kishky, A. Renduchintala, V. Chaudhary, F . Guzmán, and L. Specia, “Quality estimation without human-labeled data,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL). Online: As... on Pith
- [15] D. M. Alves, R. Rei, A. C. Farinha, J. G. C. de Souza, and A. F . T. Mar- tins, “Robust MT evaluation with sentence-level multilingual augmenta- tion,” inProceedings of the Seventh Conference on Machine Translation (WMT). Abu Dhabi, United Arab Emirates: Association for Comput... on Pith
- [16] V. Raunak, M. Post, and A. Menezes, “SALTED: A framework for SAlient long-tail translation error detection,” inFindings of the Associa- tion for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 5163–5179 on Pith
- [17] D. Kanojia, M. Fomicheva, T. Ranasinghe, F . Blain, C. Or ˘asan, and L. Specia, “Pushing the right buttons: Adversarial evaluation of quality estimation,” inProceedings of the Sixth Conference on Machine Translation (WMT). Online: Association for Computational Linguistics, Nov... on Pith
- [18] D. Saunders, “Domain adaptation and multi-domain adaptation for neural machine translation: A survey,”Journal of Artificial Intelligence Research, vol. 75, pp. 351–424, 2022 on Pith
- [19] H. Xu, Y . J. Kim, A. Sharaf, and H. H. Awadalla, “A paradigm shift in ma- chine translation: Boosting translation performance of large language models,” inProceedings of the 12th International Conference on Learn- ing Representations (ICLR), 2024, arXiv:2309.11674 on PithPith's reviewarXiv
- [20] I. Vieira, W. Allred, S. Lankford, S. Castilho, and A. Way, “How much data is enough data? fine-tuning large language models for in-house translation: Performance evaluation across multiple dataset sizes,” arXiv preprint arXiv:2409.03454v2, 2024 on PithPith's reviewarXiv
- [21] R. Aharoni, M. Johnson, and O. Firat, “Massively multilingual neural machine translation,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies (NAACL-HLT). Minneapolis, MN, USA: Asso... on Pith
show 16 more
- [22] Y . Luo, Z. Xu, Y . Ouyang, M. Y ang, D. Lin, K. Chang, T. Zheng, B. Li, P . Feng, Q. Du, T. Xiao, and J. Zhu, “NiuTrans.LMT: Toward inclusive and scalable multilingual machine translation with LLMs,” inProceed- ings of the 64th Annual Meeting of the Association for Computatio... on Pith
- [23] V. Raunak and A. Menezes, “Finding memo: Extractive memorization in constrained sequence generation tasks,” inFindings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 5153– 5162 on Pith
- [24] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F . Tramèr, and C. Zhang, “Quantifying memorization across neural language models,” inPro- ceedings of the 11th International Conference on Learning Represen- tations (ICLR), 2023, arXiv:2202.07646 on PithPith's reviewarXiv
- [25] P . H. Martins, J. Alves, P . Fernandes, N. M. Guerreiro, R. Rei, A. Farajian, M. Klimaszewski, D. M. Alves, J. Pombal, N. Boizard, M. Faysse, P . Colombo, F . Yvon, B. Haddow, J. G. C. de Souza, A. Birch, and A. F . T. Martins, “EuroLLM-9B: Technical report,” arXiv preprint a... on PithPith's reviewarXiv
- [26] M. M. Ramos, D. M. Alves, H. Gisserot-Boukhlef, J. Alves, P . H. Martins, P . Fernandes, J. Pombal, N. M. Guerreiro, R. Rei, N. Boizard, A. Farajian, M. Klimaszewski, J. G. C. de Souza, B. Haddow, F . Yvon, P . Colombo, A. Birch, and A. F . T. Martins, “EuroLLM-22B: Technical ... on PitharXiv
- [27] L. Zhu, M. Zimina-Poirot, C. Valdez, and S. Patin, “Fine-tuning NMT models and LLMs for specialised EN-ES translation using aligned cor- pora, glossaries, and synthetic data: MULTITAN at WMT25 terminol- ogy shared task,” inProceedings of the Tenth Conference on Machine Transla... on Pith
- [28] D. Zhang, Y . Khater, R. Rahli, A. Rebollo, and J. Crego, “SYSTRAN @ WMT 2025 general translation task,” inProceedings of the Tenth Con- ference on Machine Translation (WMT). Suzhou, China: Association for Computational Linguistics, 2025, pp. 599–606 on Pith
- [29] K. Guttmann, Z. Rostek, A. Charkiewicz, A. Solarski, M. Pokrywka, and A. Nowakowski, “Laniqo at WMT25 general translation task: Self- improved and retrieval-augmented translation,” inProceedings of the Tenth Conference on Machine Translation (WMT). Suzhou, China: Association f... on Pith
- [30] European Parliament and Council of the European Union, “Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on machinery, and amending Directive 95/16/EC (recast),” Official Journal of the European Union, L 157, pp. 24–86, Jun. 2006, 9 June 2006. ... on Pith
- [31] ——, “Regulation (EU) 2023/1230 of the European Parliament and of the Council of 14 June 2023 on machinery and repealing Directive 2006/42/EC of the European Parliament and of the Council and Council Directive 73/361/EEC,” Official Journal of the European Union, L 165, pp. 1–10... on Pith
- [47] M. Chen, A. Reneau, and K. Flansburg, “Introducing Clef: our open-source decision models, and new RL fine-tuning platform,” Cloudflare Blog, Oct. 2026, accessed: 2026-10-17. [Online]. Available: https://blog.cloudflare.com/clef-decision-models on Pith
- [48] Cloudflare, “Clef,” Model card, Hugging Face, 2026, accessed: 2026-10-17. [Online]. Available: https://huggingface.co/cloudflare/clef on Pith
- [49] F . Lukassen, C. Weisser, and A. Silbersdorff, “Cheap, consistent, calibrated, but biased? Typed decision models as judges,” arXiv preprint arXiv:2610.18122, 2026. [Online]. Available: https://arxiv.org/ abs/2610.18122 on PithPith's reviewarXiv
- [50] M. Popovi ´c, “chrF: Character n-gram F-score for automatic MT eval- uation,” inProceedings of the Tenth Workshop on Statistical Machine Translation (WMT). Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 392–395 on Pith
- [52] J. Gun and T. Oksanen, “Agri-Query: A case study on RAG vs. long-context LLMs for cross-lingual technical question answering,” arXiv:2508.18093, 2025. [Online]. Available: https://arxiv.org/abs/2508. 18093 Technical University of Munich Chair of Agrimechatronics 15 ISBN:978-3-... on PitharXiv
- [2025] [Online]. Available: https://arxiv.org/abs/2512.09434 on PitharXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment