REVIEW 31 cited by
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator
Forward citations
Cited by 31 Pith papers
-
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
LLMs underperform on culturally loaded Chinese-to-Japanese translation, while both human and automatic evaluation become unstable when culture is the main variable.
-
ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
A new open-source benchmark evaluates LLM-generated end-to-end ML pipelines from Kaggle competition descriptions translated into 13 languages, with 6 private tasks to limit data leakage.
-
Mind the Language Gap in Digital Humanities: LLM-Aided Translation of SKOS Thesauri
WOKIE combines external translation services with LLM-based refinement to translate SKOS thesauri, improving translation quality and boosting ontology matching F1 scores.
-
Decoding Machine Translationese in English-Chinese News: LLMs vs. NMTs
Machine-translated English-to-Chinese news differs from original Chinese news in measurable ways, and LLM and NMT outputs can be partially but not fully distinguished by linguistic features.
-
Can Peter Pan Survive MT? A Stylometric Study of LLMs, NMTs, and HTs in Children's Literature Translation
LLM translations of Peter Pan sit stylistically closer to human translations than NMT outputs do on several child-literature features, but the prompting strategy and possible training-data overlap partly explain the c...
-
BTPD: A Multilingual Hand-curated Dataset of Bengali Transnational Political Discourse Across Online Communities
The paper presents BTPD, a new multilingual dataset of 2,235 hand-curated Bengali political posts from three online platforms, along with a descriptive topic overview.
-
Exploring In-context Example Generation for Machine Translation
DAT generates query-specific in-context translation examples using only an LLM, improving English-to-low-resource translation over zero-shot in most tested languages.
-
Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages
Users who first used a Spanish AI writing assistant subsequently used the English AI writing assistant less, suggesting a spillover that violates choice independence.
-
Addressing speaker gender bias in large scale speech translation systems
Fine-tuning a large speech translation model on GPT-4-reformulated gender-balanced training data raises MuST-SHE feminine-form accuracy from about 10% to over 84% without BLEU loss.
-
Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset
Motion-X++ provides 19.5M 3D whole-body pose annotations across 120.5K sequences with text, audio, video, and motion modalities.
-
Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models
Adaptive selection of translation examples through a hybrid LLM-embedding retriever and a self-supervised reranker improves few-shot machine translation across several LLMs.
-
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.
-
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
On Swiss legal translation, reinforcement learning with a ChrF reward improves small open models more than supervised fine-tuning, but frontier reasoning models still score higher.
-
$M^2PO$: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation
M2PO combines a QE-plus-alignment reward with a dynamic curriculum and multi-pair DPO loss, and reports WMT21-22 gains for a 7B translation model, but the abstract's WMT23/24 9B parity claims are unsupported.
-
ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts
An empirical study of developer-ChatGPT refactoring chats yields a 25-theme taxonomy, apology/affirmation signals, and a structured prompt template that reduces conversation turns.
-
Hunyuan-MT Technical Report
Hunyuan-MT and Chimera, a 7B open-source translation model and its multi-candidate fusion variant, claim state-of-the-art multilingual translation including Mandarin to minority languages, with open weights.
-
Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters
A 7B open-weight translation model matches or outperforms far larger commercial systems across 28 languages in automatic and human evaluations.
-
Psychology-Driven Enhancement of Humour Translation
A decomposition-and-recomposition prompt method for humor translation reports large gains on LLM-based metrics, but the evaluation lacks human validation and statistical checks.
-
Information Loss in LLMs' Multilingual Translation: The Role of Training Data, Language Proximity, and Language Family
Round-trip translation quality in GPT-4 and Llama 2 is jointly shaped by training data volume and language distance from English, with orthographic, phylogenetic, syntactic, and geographic distances as the strongest p...
-
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study
A new data-mixing recipe (Parallel-First Monolingual-Second) and a 9B model, GemmaX2-28, achieve translation quality competitive with Google Translate and GPT-4 across 28 languages.
-
Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History
On a translated Lithuanian history multiple-choice benchmark, GPT-4o beats all tested open and Nordic-tuned models, and Nordic-language fine-tuning does not improve accuracy.
-
Self-Evolution Knowledge Distillation for LLM-based Machine Translation
A token-adaptive distillation method that mixes teacher and ground-truth targets only for hard tokens yields consistent BLEU gains in LLM translation.
-
Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation
A framework that generates images from source sentences with a Stable Diffusion model fine-tuned by a scene-graph reward, then feeds them into a multimodal LLM, is claimed to improve machine translation, but key compa...
-
Exploring and Mitigating Fawning Hallucinations in Large Language Models
A contrastive decoding method that contrasts a misleading prompt against a neutral rewrite reduces fawning hallucinations in LLMs, though most of the gain comes from the neutral prompt itself.
-
Bangla-Bayanno: A 52K-Pair Bengali Visual Question Answering Dataset with LLM-Assisted Translation Refinement
A 52,650-pair Bengali VQA dataset built by translating VQA v2 with GPT-4, claimed as the largest open-source Bangla benchmark but weakly validated.
-
Automated Consistency Analysis of LLMs
LLMs frequently produce inconsistent answers to repeated cybersecurity questions, but the paper's validation methods rely on the models judging their own and each other's responses.
-
Can ChatGPT Diagnose Alzheimer's Disease?
On 9,300 ADNI records, multi-shot prompting with combined MRI and cognitive features let ChatGPT reach 94.6% accuracy, but missing baselines and possible data leakage limit the claim.
-
Application of Multimodal Large Language Models in Autonomous Driving
A fine-tuned CogVLM2 with chain-of-thought is applied to autonomous driving tasks, but its claimed performance gains are not supported by the reported quantitative results.
-
Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement
Marco-LLM, a Qwen2-based model continually pretrained on 300B multilingual tokens, reports large gains on low-resource language benchmarks and any-to-any translation.
-
Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.
-
Semiotic Reconstruction of Destination Expectation Constructs An LLM-Driven Computational Paradigm for Social Media Tourism Analytics
Using GPT-4 to label Chinese travel posts, the study claims leisure and social expectations predict social media engagement more than nature or emotional expectations.
Discussion (0). Continue with ORCID to comment.