{"id":"bbb77243-8c9e-433c-9b2b-a7fcd71efaf3","arxiv_id":"2605.31452","paper_version":1,"verdict":"UNVERDICTED","confidence":"UNKNOWN","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Local LLMs via Ollama match or exceed some local NMT systems and a frontier LLM on a new multilingual corpus but lag behind top commercial NMTs like DeepL.","lead":"This paper benchmarks local LLMs running offline via Ollama for translation on an expanded multilingual corpus, finding the best ones match some local NMT systems but trail top commercial tools. Smart generalists might read it to assess practical offline options for privacy-sensitive freelance translation work.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"MATEO automatic scores on RFMC corpus treated as proxy for professional quality without human validation or domain checks","rationale":"The reader's weakest assumption directly identifies the same unverified proxy step; confirming or refuting it via the proposed human check would move the paper from UNVERDICTED to CONDITIONAL (or stronger) depending on outcome. No other internal inconsistency appears load-bearing from the given description.","tokens_in":1736,"tokens_out":322,"duration_ms":11118,"concrete_test":"Randomly sample 100 sentences from the evaluated RFMC set; obtain blind professional human ratings (adequacy, fluency, terminology) for outputs of the top local LLM, best local NMT, and top commercial NMT; recompute rankings and check whether MATEO ordering is preserved (Spearman rho > 0.7) or reversed on any pair.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim (best local LLMs match/surpass local NMT and GPT-5.2) rests on MATEO scores from 1000+ sentences under single-prompt, no-adaptation conditions. MATEO (like other reference-based metrics) is known to correlate only moderately with human judgments on semantic fidelity, terminology precision, and stylistic appropriateness in specialized domains; the paper reports no human evaluation, no error analysis, and no correlation study between MATEO and professional raters on this corpus. The RFMC expansion adds German/Chinese but does not demonstrate domain representativeness for confidentiality-sensitive content.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper expands the Reeve Foundation Trilingual Corpus (RFTC) into the multilingual RFMC by adding German and Simplified Chinese references. It benchmarks local LLMs runnable via Ollama on 1000+ sentences across four language directions using consistent single-prompt calls and MATEO automatic evaluation, comparing them to commercial NMTs (DeepL, Baidu), GPT-5.2, and local NMT systems (OPUS-CAT, etc.). The results indicate that the best local LLMs match or surpass local NMT and the frontier LLM but remain behind top commercial NMTs, supporting their use in confidentiality-sensitive translation workflows.","tokens_in":1861,"tokens_out":481,"duration_ms":17966,"significance":"If the findings hold, the work offers practical value for freelancers and small providers needing offline translation tools under privacy constraints. The expansion of the corpus to include German and Chinese, the consistent benchmarking protocol across models and metrics, and the direct empirical comparisons are notable strengths that could guide technology adoption in specialized domains.","major_comments":[{"comment":"Abstract / Results: The central claim that best local LLMs match or surpass local NMT systems and GPT-5.2 rests on MATEO scores from 1000+ sentences under single-prompt conditions. No human evaluation, error analysis, or correlation between MATEO and professional judgments on the RFMC corpus is reported, which is load-bearing because reference-based metrics are known to correlate only moderately with human assessments of terminology precision and stylistic appropriateness in specialized domains.","section":"Abstract / Results"},{"comment":"Abstract / Methods: Sentence selection criteria for the 1000+ sentences, statistical testing of MATEO score differences, and checks for domain representativeness of the RFMC expansion for confidentiality-sensitive content are not described, undermining assessment of the robustness of the cross-model and cross-language comparisons.","section":"Abstract / Methods"}],"minor_comments":[{"comment":"Abstract: The transition from 'Reeve Foundation Trilingual Corpus (RFTC)' to 'multilingual corpus (RFMC)' is introduced clearly, but verify consistent acronym usage and full expansion on first mention in all sections.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on arXiv:2605.31452. We address each major comment below with planned revisions to improve methodological transparency and acknowledge limitations of the automatic evaluation.","responses":[{"response":"We agree that human evaluation would strengthen interpretation of the results for specialized domains. The manuscript centers on a reproducible automatic protocol using MATEO for practical benchmarking accessible to freelancers. We will add a Limitations section noting the reliance on automatic metrics and their moderate correlation with human judgments, plus a sample-based qualitative error analysis to illustrate patterns in the outputs.","revision_made":"partial","referee_comment":"[Abstract / Results] Abstract / Results: The central claim that best local LLMs match or surpass local NMT systems and GPT-5.2 rests on MATEO scores from 1000+ sentences under single-prompt conditions. No human evaluation, error analysis, or correlation between MATEO and professional judgments on the RFMC corpus is reported, which is load-bearing because reference-based metrics are known to correlate only moderately with human assessments of terminology precision and stylistic appropriateness in specialized domains."},{"response":"We accept that these details require expansion for full assessment of robustness. The revised manuscript will expand the Methods section to specify sentence selection criteria, include statistical significance testing on MATEO differences, and discuss domain representativeness of the RFMC expansion drawing on the Reeve Foundation source materials.","revision_made":"yes","referee_comment":"[Abstract / Methods] Abstract / Methods: Sentence selection criteria for the 1000+ sentences, statistical testing of MATEO score differences, and checks for domain representativeness of the RFMC expansion for confidentiality-sensitive content are not described, undermining assessment of the robustness of the cross-model and cross-language comparisons."}],"tokens_in":1438,"tokens_out":392,"duration_ms":20523,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a practical benchmark for freelancers who need offline translation on sensitive material. They expand the prior Reeve Foundation corpus to include German and Chinese, run over 1000 sentences through several Ollama models in four directions, and compare them to DeepL, Baidu, GPT-5.2, and local NMT tools using MATEO. The best local models come out ahead of some local NMT systems and the frontier LLM on the automatic scores, though still behind the top commercial engines.\n\nWhat works is the consistent single-prompt protocol and the direct head-to-head numbers. It gives freelancers concrete data on model size and language-pair variation without requiring fine-tuning or cloud access. The focus on confidentiality constraints is a real use case that standard MT papers often skip.\n\nThe soft spot is the evaluation. Everything rests on MATEO without any human raters, error analysis, or correlation check against professional judgments on this corpus. Reference-based metrics like MATEO are known to miss terminology precision and stylistic fit in specialized domains, and the paper does not test domain adaptation or sentence selection criteria for confidential content. The German and Chinese additions are welcome but do not address how representative the sentences are for actual privacy-sensitive work.\n\nThis is for translators and small LSPs who need offline options and want to see which local models are worth testing themselves. A reader in that niche will find usable numbers. It is worth sending to peer review because the empirical setup is straightforward and the application is clear, though referees will almost certainly request human evaluation to strengthen the quality claims.","headline":"The paper expands their RFMC corpus and shows some local LLMs competitive with local NMT on MATEO scores for privacy-focused translation, but the claims rest on automatic metrics alone.","tokens_in":2338,"tokens_out":402,"would_cite":false,"duration_ms":12158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Selected local LLMs match or surpass local NMT systems and a frontier model for confidential translation but remain behind top commercial NMTs.","keywords":["local LLMs","machine translation","confidential translation","offline translation","benchmarking","multilingual corpus","automatic evaluation","freelance translation"],"falsifier":"A side-by-side human evaluation by professional translators on the same 1000+ sentence outputs that produces a different ranking between the top local LLMs and the local NMT systems.","tokens_in":2637,"feed_emoji":"📊","tokens_out":751,"duration_ms":14296,"temperature":0.7,"pith_summary":"The paper expands an existing trilingual corpus into a multilingual one with German and Chinese references and then runs a consistent benchmark of locally runnable LLMs on over one thousand sentences across four language directions. Outputs are generated with single-prompt calls through Ollama and scored automatically with MATEO against commercial NMT engines, a frontier LLM, and professional-grade local NMT systems. The central finding is that the strongest local LLMs reach or exceed the local NMT systems and the frontier model while still trailing the leading commercial tools. This matters for freelance translators and small providers who must keep sensitive material offline and cannot rely on cloud services. A sympathetic reader sees a practical route to evaluate and select offline models without requiring fine-tuning or domain adaptation.","feed_headline":"Best local LLMs match local NMT systems for confidential work","feed_subtitle":"Benchmark on expanded RFMC corpus shows top offline models rival dedicated local tools but trail commercial leaders across four language dir","key_machinery":"The RFMC corpus paired with MATEO automatic scoring applied to single-prompt outputs from Ollama-hosted local LLMs, used to rank performance against commercial and local NMT baselines.","core_discovery":"Building on prior work, the authors expand the Reeve Foundation Trilingual Corpus into the RFMC by adding sentence-aligned German and Simplified Chinese references. They then benchmark several locally runnable models via Ollama on 1000+ sentences in four language directions using single-prompt calls with no fine-tuning. Automatic evaluation with MATEO shows substantial variation by language pair and model size; the best local LLMs match or surpass local NMT systems and a frontier LLM while remaining behind top commercial NMTs such as DeepL and Baidu.","pith_inferences":["The same benchmark protocol could be applied to domain-specific corpora drawn from legal or medical texts to test whether the observed ranking holds under stricter terminology demands.","Integration of the highest-scoring local LLMs into existing desktop tools such as OPUS-CAT might reduce post-editing effort for users already familiar with those interfaces.","If model size continues to improve local multilingual capability, the current gap to commercial NMTs could narrow without any increase in data exposure."],"forward_implications":["Freelance translators gain concrete evidence that selected local LLMs can serve as viable offline options for privacy-constrained assignments.","Performance differences across language directions and model sizes guide model selection without additional training.","The benchmark method itself supplies a low-barrier template that smaller providers can replicate on their own data.","Results highlight the remaining gap to commercial leaders and the potential value of further scaling local models."],"fun_headline_variants":["Local LLMs match local NMT in RFMC confidential benchmarks","RFMC tests local LLMs for confidential translation workflows","Local LLMs benchmarked on RFMC against NMT and GPT models","Offline LLM results vary by language direction on RFMC corpus"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Automatic evaluation with MATEO on general-domain sentences from the expanded corpus serves as a reliable stand-in for professional translation quality in confidentiality-sensitive work without human assessment or domain adaptation.","fun_headline_variants_meta":{"raw":{"variants":["Local LLMs match local NMT in RFMC confidential benchmarks","RFMC tests local LLMs for confidential translation workflows","Local LLMs benchmarked on RFMC against NMT and GPT models","Offline LLM results vary by language direction on RFMC corpus"]},"model":"grok-4.3","cost_usd":0.008478,"raw_usage":{"total_tokens":3782,"prompt_tokens":728,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":84778000,"prompt_tokens_details":{"text_tokens":728,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2986,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":728,"tokens_out":68,"duration_ms":18808,"temperature":1.0,"reasoning_tokens":2986,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T22:47:26.885715+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A side-by-side human evaluation by professional translators on the same 1000+ sentence outputs that produces a different ranking between the top local LLMs and the local NMT systems.","supporting_citations":[],"review_version":1}