Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation

Josef van Genabith; Marcos Zampieri; Mihaela Vela; Santanu Pal; Sudip Kumar Naskar

REVIEW 4 major objections 5 minor 24 references

Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Machine translation output becomes the starting point in about 80% of translator choices.

desk verdict A small, clearly reported CAT tool study whose headline 80% MT selection rate rests on three translators and a post-hoc design change; still worth a look for the color-coding and log data. read the letter →

arxiv 1908.06140 v1 pith:IOBWQFLT submitted 2019-08-16 cs.CL

Mihaela Vela , Santanu Pal , Marcos Zampieri , Sudip Kumar Naskar , Josef van Genabith This is my paper · ORCID

classification cs.CL

keywords computer-aidedtranslationmemorymachinepost-editingautomaticcolorcodingCATtoolevaluationEnglish-German

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a web-based CAT tool can be improved by changing how translation suggestions are retrieved and presented, and that with these changes professional translators overwhelmingly take machine-translation output as their starting point. The authors report that in an English-to-German user study with three professional translators, the MT suggestion was selected for about 80% of segments, with translation-memory suggestions and translation from scratch making up the rest. They argue this shows a clear preference for MT as the basis of post-editing, even though the three translators did not agree with each other on which segments to choose MT. They also report that translators valued a color-coding scheme that highlights matching and mismatched parts of TM suggestions, helping them decide what to edit.

What carries the argument

The central object is CATaLog Online, a web-based CAT tool whose suggestion pipeline combines three engines: a TM retriever that scores matches with a TER-based alignment and Needleman-Wunsch-style match/reward edit scoring, an integrated statistical MT system, and an automatic post-editing (APE) system built on an operation sequence model. Lucene/Nutch retrieval is used to keep TM search fast on large memories. The mechanism that carries the empirical claim is the suggestion-selection design: the tool presents MT, TM, APE, and from-scratch options side by side, logs which engine was chosen, and color-codes TM suggestions so matched text is green and mismatched text red. In the final experiment, APE was removed because its output was too similar to MT, leaving translators to choose between MT, TM, and scratch; the logged choices are the direct evidence for the roughly 80% MT selection rate.

What would settle it

Run the same English-to-German task in a CAT tool with APE included, with MT, TM, and from-scratch options presented in randomized order, and with more than three professional translators; if the MT selection share falls well below 80% or splits evenly across options, the claimed clear MT preference is not supported.

Watch

Extended reading notes

Core claim

The paper's central discovery is the observed selection behavior in the CATaLog Online environment: when professional translators are offered MT output, a TM suggestion, or translation from scratch, they chose MT in roughly 160 of 200 sentences (about 80%), and this held over both the full set and the 100-sentence common set. The authors interpret this as evidence that MT should be treated as the primary post-editing starting point in CAT tools, with TM and scratch translation as fallbacks. They also found that translators did not reliably agree on which segments should use MT, and that editing time was translator-dependent and only weakly correlated with edit count. A supporting finding is that the interface's color coding of matched and mismatched words in TM suggestions was positively received, and was credited with making TM suggestion selection easier.

Load-bearing premise

The load-bearing premise is that the three professional translators, tested after the APE option was removed because its output resembled MT too closely, give a representative picture of translator preference; if a small, modified study misrepresents that preference, the roughly 80% MT selection claim weakens.

Editorial extensions

If this is right

If MT is the preferred starting point, CAT-tool interfaces should lead with MT output and treat TM as a secondary suggestion rather than the default.
Color-coding the matched and mismatched parts of TM suggestions gives translators a quick visual signal of which parts need editing, which can reduce the effort of deciding among suggestions.
Because editing time varies by translator and correlates weakly with the number of edits, post-editing time alone is not a reliable productivity metric in this setting.
APE output that closely resembles MT output adds little distinguishable value to the translator's choice set, supporting the decision to leave it out of the final evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported ~80% selection rate was measured after APE was removed and with only three translators; if the experiment were repeated with APE included and suggestion order randomized, the MT share might be lower, since translators were forced to choose between MT and a very similar APE output in the earlier student study.
The dominance of MT may reflect the quality of the integrated MT system and the news-domain data; for domains where TM consistency matters more than fluency, translators might favor TM suggestions, so the preference should not be generalized without domain variation.
A testable extension would be an ablation of the color-coding feature: give translators the same MT and TM options with and without highlighting, and measure whether selection and post-editing speed change.
Because translators did not agree with each other on which segments should use MT, a per-segment MT-quality estimator could help by pre-selecting MT suggestions only where they are likely to be accepted, shifting from a global MT default to a segment-level recommendation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a modest systems contribution: it describes improvements to CATaLog Online, an existing web CAT tool, and reports two small user studies. The genuinely new result is the second study, where three professional translators chose the MT output about 80% of the time, with very low inter-translator agreement. That specific number isn't in the authors' prior work, and it is consistent with earlier findings that translators tend to prefer MT as a starting point.

The paper deserves credit for being transparent about its own design change. When the student pilot showed that MT and APE outputs were too similar to choose between reliably, the authors dropped APE and reran the study with professionals. That is honest reporting. The color-coding scheme for TM matches is a sensible idea, and the positive feedback from the three translators is a reasonable signal. The detailed XML logs are also useful for anyone doing translation process research.

The soft spots are the ones the reader and stress-test highlight. Removing APE because it is nearly indistinguishable from MT means the 80% figure measures MT only because the near-identical alternative is absent. A substantial share of those 80% may represent "machine output" generally rather than MT specifically. The paper says this clearly, but it does not solve the problem. With n=3 and no significance tests or confidence intervals, the word "clear" is doing more work than the data can support. Also, the kappa section contains a small but confusing inconsistency: the text says number of edits has low κ and editing time has high κ, while Table 3 shows the opposite (editing time near zero or negative, number of edits around 0.3–0.5). That should be fixed.

Proportionally, these are weaknesses, not fatal flaws. The central observation is internally consistent with Table 2, and the paper does not oversell beyond its evidence. It lists limitations and frames the findings as preliminary.

Who is this for? Researchers and tool-builders working on CAT interfaces, TM/MT integration, and translation process effort. A reader in that space gets a small, honest data point plus the color-coding idea.

Recommendation: send it to peer review. It's not a strong paper, but it is a legitimate empirical note with a real result and a clearly documented method. A serious referee should see it, and with revisions the kappa mix-up and the MT-vs-APE interpretation issue are fixable.

Referee Report

4 major / 5 minor

Summary. The paper presents CATaLog Online, a web-based CAT tool offering translation suggestions from translation memories (TM), machine translation (MT), and automatic post-editing (APE), together with color-coded intra-segment feedback and a Lucene-based retrieval backend. The authors report two user studies: a comparison of post-editing time between MateCat and CATaLog Online with 16 translation students, and a choice experiment in which three professional translators selected between MT, TM, and translating from scratch over 200 English-to-German news sentences. The central empirical claim is that the MT system achieves a selection rate of around 80% and that translators show a clear preference for MT output as the starting point for their translations. The paper also reports inter-rater agreement statistics, edit-type distributions, and qualitative user feedback on the tool's usability.

Significance. If the central claim were robust, the result would be a useful data point on translator behavior in CAT environments, namely that professional translators, when given MT, TM, and scratch options, predominantly choose MT. The paper's strength is its concrete system description: CATaLog Online is a real, web-based, open-source tool with logging capabilities and a color-coding scheme that is a plausible usability contribution. The authors also make an honest attempt to measure inter-rater agreement with Cohen's kappa, and they disclose negative feedback and limitations. However, the headline empirical claim rests on only three professional translators after the APE condition was removed post hoc, and no significance tests or confidence intervals are provided. The study therefore currently supports a much weaker statement: in a three-option design without APE, the three participating translators chose MT for roughly 80% of the sentences. The paper's significance as a system description is real, but its evaluative conclusions need substantial qualification.

major comments (4)

[Section 4, Table 2] The headline claim that the MT system achieves a selection rate of around 80% is based on data from only three professional translators (T1, T2, T3), and the paper reports no confidence intervals or significance tests. With n=3, the authors' own Table 3 shows that inter-rater agreement on suggestion selection is very low (pairwise Cohen's kappa between 0.05 and 0.20), which undercuts the statement in Sections 4 and 5 that 'translators have a clear preference in choosing the output of the MT system.' The claim should be reworded as a descriptive observation about these three translators, not a generalizable preference.
[Section 4, APE exclusion] The APE condition was removed because, in the student experiment, 'the MT output and the output from the APE system are very similar to each other.' This means the reported 80% MT selection rate was measured in a design where no near-identical alternative was available. The data cannot distinguish a preference for MT specifically from a preference for machine-generated output, since a large share of the 80% might have gone to APE had it been offered. The abstract and conclusion credit 'the MT system' without acknowledging this design modification; the authors should either reframe the claim as 'MT output, in the absence of a near-duplicate APE option,' or provide professional-translator data from a four-option condition.
[Table 1, MateCat comparison] The first user study shows that CATaLog Online was slower than MateCat for all ten common sentences (for example, Stud1 took 1112 seconds in MateCat versus Stud9's 3079 seconds in CATaLog Online), and the paper does not report any statistical analysis of this difference. The paper's positive conclusions about user preference and the tool's usefulness do not directly address this performance gap. If the paper claims to 'improve CAT tools,' it should either temper the productivity-related claims or provide an analysis of the time difference, for instance by showing that the extra time is accompanied by better quality or a deliberate interface trade-off.
[Section 4, Table 3 and correlation] The agreement analysis reports Cohen's kappa and Pearson's rho without confidence intervals, and with only three raters the kappa estimates in Table 3 are highly unstable. The sentence on Pearson's rho is also confusing: the text says the authors tested whether the total number of edits (with a low kappa) influences post-editing time (with a high kappa), but the kappa values in Table 3 do not cleanly separate these variables. This analysis should be either clarified with appropriate uncertainty measures or omitted from the main claims.

minor comments (5)

[Section 3, first paragraph] There is a typo: 'CATaLog Onlineof-fers' should be 'CATaLog Online offers.'
[Section 1, CATaLog description] The phrase 'enhanced with with a new interface layout' contains a duplicated 'with.'
[Table 2 caption] The caption reads '200 sentenced' and should read '200 sentences.'
[Section 4, user feedback] The user feedback section is anecdotal and does not describe a systematic procedure for collecting or coding the positive and negative impressions; adding a short method note would help readers interpret these results.
[Section 4.2, limitations] The third limitation bullet is acknowledged but not connected to the main 'clear preference' claim; the authors should state explicitly that the current experiment cannot assess whether choosing MT output improves final translation quality, coherence, or cohesion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 80% MT-selection claim is an observed log-based measurement, not a derived or fitted result.

full rationale

The paper makes no formal derivation whose conclusion is equivalent to its inputs. Its central claim is an empirical observation: professional translators selected the MT suggestion in roughly 80% of segments, as recorded in CATaLog Online's logs and reported in Table 2. This is a measurement, not a prediction obtained from fitted parameters, and no equation in the paper defines the reported preference in terms of the study's own setup. The self-citations to prior work by the authors (Nayek et al. 2015; Pal et al. 2015a,b; Pal et al. 2016a,b) are used to identify the provenance of the TM, MT, and APE components integrated into the tool, not to justify the observed user behavior. Likewise, the exclusion of the APE option because 'the MT output and the output from the APE system are very similar to each other' is an explicitly stated design change that may limit the interpretation of the 80% rate as MT-specific rather than machine-output-specific, but it is a methodological limitation, not a circular reduction. The paper also includes an explicit limitations section, and nothing there asserts that a result is derived from an assumption identical to itself. No uniqueness theorem, ansatz smuggling, or renaming of a known result is present. The central claim is therefore self-contained empirical evidence, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is a systems and user-study paper with no mathematical derivation and no fitted parameters. The claims rest on the validity of the user study design and on prior work by the same authors for the tool's core components.

assumptions (4)

domain assumption Post-editing time is a useful measure for comparing CAT tools.
Used for the MateCat versus CATaLog Online comparison in Table 1, though the paper later argues that time is a subjective measure of effort.
domain assumption Selection of a suggestion by a translator indicates preference for that suggestion.
The central MT preference result is based on selection counts in Table 2, assuming that choices are deliberate rather than arbitrary.
domain assumption The color-coding scheme improves decision making.
Supported only by self-reported feedback in Section 4.1, not by a controlled comparison.
domain assumption The TER plus Needleman-Wunsch retrieval metric is sound.
Invoked via citation to Nayek et al. (2015) without direct evaluation in this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation." pith.science (2026). https://pith.science/paper/IOBWQFLT

@misc{pith2026190806140,
  author       = {Pith},
  title        = {Pith review of: Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOBWQFLT}},
  note         = {Machine review of arXiv:1908.06140}
}

read the original abstract

This paper describes strategies to improve an existing web-based computer-aided translation (CAT) tool entitled CATaLog Online. CATaLog Online provides a post-editing environment with simple yet helpful project management tools. It offers translation suggestions from translation memories (TM), machine translation (MT), and automatic post-editing (APE) and records detailed logs of post-editing activities. To test the new approaches proposed in this paper, we carried out a user study on an English--German translation task using CATaLog Online. User feedback revealed that the users preferred using CATaLog Online over existing CAT tools in some respects, especially by selecting the output of the MT system and taking advantage of the color scheme for TM suggestions.

Figures

Figures reproduced from arXiv: 1908.06140 by the authors.

**Figure 1.** Correlation between the overall number of edits and edit time. and APE. The tool is specifically designed to improve post-editing productivity and user experience with CAT. A novel feature in the tool is a new intra-segment color coding scheme that highlights matching and irrelevant fragments in suggested TM segments. The feedback from the translators show that color coding the TM suggestions makes the decision… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 23 canonical work pages

[1]

Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., Negri, M., Neveol, A., Neves, M., Popel, M., Post, M., Rubino, R., Scarton, C., Specia, L., Turchi, M., Verspoor, K., and Zampieri, M. (2016). Findings of the 2016 Conference on Machine Translation . In Proceedings of WMT

work page 2016
[2]

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales . Educational and Psychological Measurement , 20(1):37--46

work page 1960
[3]

Federico, M., Bertoldi, N., Cettolo, M., Negri, M., Turchi, M., Trombetti, M., Cattelan, A., Farina, A., Lupinetti, D., Martines, A., et al. (2014). The Matecat Tool . In Proceedings of COLING

work page 2014
[4]

Guerberof, A. (2012). Productivity and Quality in the Post-Edition of Outputs from Translation Memories and Machine Translation . PhD thesis, Rovira and Virgili University Tarragona

work page 2012
[5]

Gupta, R., Or a san, C., Zampieri, M., Vela, M., van Genabith, J., and Mitkov, R. (2016). Improving Translation Memory Matching and Retrieval Using Paraphrases . Machine Translation , 30(1):19--40

work page 2016
[6]

Gupta, R., Or a san, C., Zampieri, M., Vela, M., and van Genabith, J. (2015). Can Translation Memories Afford not to Use Paraphrasing? In Proceedings of EAMT

work page 2015
[7]

Herbig, N., Pal, S., Vela, M., Kr \"u ger, A., and van Genabith, J. (2019). Multi-modal Indicators for Estimating Perceived Cognitive Load in Post-editing of Machine Translation . Machine Translation

work page 2019
[8]

Koponen, M. (2012). Comparing Human Perceptions of Post-editing Effort with Post-editing Operations . In Proceedings of WMT

work page 2012

Show all 24 references

[9]

Koponen, M. (2016). Is machine Translation Post-editing Worth the Effort? A Survey of Research into Post-editing and Effort . Journal of Specialised Translation , 25:131--141

2016
[10]

and O’ B rien, S

Moorkens, J. and O’ B rien, S. (2017). Assessing User Interface Needs of Post-editors of Machine Translation . In Human Issues in Translation Technology , pages 127--148. Routledge

2017
[11]

K., Pal, S., Zampieri, M., Vela, M., and van Genabith, J

Nayek, T., Naskar, S. K., Pal, S., Zampieri, M., Vela, M., and van Genabith, J. (2015). CATaLog: New Approaches to TM and Post Editing Interfaces . In Proceedings of NLP4TM

2015
[12]

O' B rien, S. (2012). Translation as Human–computer Interaction . Translation Spaces , 1:101--122

2012
[13]

Pal, S., Naskar, S., and van Genabith, J. (2015a). UdS-Sant: English--German Hybrid Machine Translation System . In Proceedings of WMT

2015
[14]

K., and van Genabith, J

Pal, S., Vela, M., Naskar, S. K., and van Genabith, J. (2015b). USAAR-SAPE: An English--Spanish Statistical Automatic Post-Editing System . In Proceedings of WMT

2015
[15]

K., Nayak, T., Vela, M., and van Genabith, J

Pal, S., Zampieri, M., Naskar, S. K., Nayak, T., Vela, M., and van Genabith, J. (2016a). CATaLog Online: Porting a Post-editing Tool to the Web . In Proceedings of LREC

2016
[16]

Pal, S., Zampieri, M., and van Genabith, J. (2016b). USAAR: An Operation Sequential Model for Automatic Statistical Post-editing . In Proceedings of WMT

2016
[17]

Schneider, D., Zampieri, M., and van Genabith , J. (2019). Translation Memories and the Translator: A Report on a User Survey . Babel , pages 734--762

2019
[18]

van den Bergh, J., Geurts, E., Degraen, D., Haesen, M., van der Lek-Ciudin, I., and Coninx, K. (2015). Recommendations for Translation Environments to Improve Translators' Workflows . In Proceedings of Translating and the Computer

2015
[19]

Vieira, L. (2014). Indices of Cognitive Effort in Machine Translation Post-editing . Machine Translation , 3(28):187--216

2014
[20]

and Vela, M

Zampieri, M. and Vela, M. (2014). Quantifying the Influence of MT Output in the Translators’ Performance: A Case Study in Technical Translation . In Proceedings of HaCaT

2014
[21]

C., and Seghiri, M

Zaretskaya, A., Pastor, G. C., and Seghiri, M. (2015). Translators’ Requirements for Translation Technologies: Results of a User Survey . In Proceedings of New Horizons is Translation and Interpreting Studies

2015
[22]

C., and Seghiri, M

Zaretskaya, A., Vela, M., Pastor, G. C., and Seghiri, M. (2016a). Comparing Post-Editing Difficulty of Different Machine Translation Errors in Spanish and German Translations from English . International Journal of Language and Linguistics , 3(3)

2016
[23]

C., and Seghiri, M

Zaretskaya, A., Vela, M., Pastor, G. C., and Seghiri, M. (2016b). Measuring Post-editing Time and Effort for Different Machine Translation Errors . New Voices in Translation Studies , 15:63--92

2016
[24]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., Negri, M., Neveol, A., Neves, M., Popel, M., Post, M., Rubino, R., Scarton, C., Specia, L., Turchi, M., Verspoor, K., and Zampieri, M. (2016). Findings of the 2016 Conference on Machine Translation . In Proceedings of WMT

work page 2016

[2] [2]

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales . Educational and Psychological Measurement , 20(1):37--46

work page 1960

[3] [3]

Federico, M., Bertoldi, N., Cettolo, M., Negri, M., Turchi, M., Trombetti, M., Cattelan, A., Farina, A., Lupinetti, D., Martines, A., et al. (2014). The Matecat Tool . In Proceedings of COLING

work page 2014

[4] [4]

Guerberof, A. (2012). Productivity and Quality in the Post-Edition of Outputs from Translation Memories and Machine Translation . PhD thesis, Rovira and Virgili University Tarragona

work page 2012

[5] [5]

Gupta, R., Or a san, C., Zampieri, M., Vela, M., van Genabith, J., and Mitkov, R. (2016). Improving Translation Memory Matching and Retrieval Using Paraphrases . Machine Translation , 30(1):19--40

work page 2016

[6] [6]

Gupta, R., Or a san, C., Zampieri, M., Vela, M., and van Genabith, J. (2015). Can Translation Memories Afford not to Use Paraphrasing? In Proceedings of EAMT

work page 2015

[7] [7]

Herbig, N., Pal, S., Vela, M., Kr \"u ger, A., and van Genabith, J. (2019). Multi-modal Indicators for Estimating Perceived Cognitive Load in Post-editing of Machine Translation . Machine Translation

work page 2019

[8] [8]

Koponen, M. (2012). Comparing Human Perceptions of Post-editing Effort with Post-editing Operations . In Proceedings of WMT

work page 2012