Pith. sign in

REVIEW 2 major objections 3 minor 184 cited by

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

T0 review · 2 major / 3 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GNMT, a deep LSTM neural machine translation system with wordpieces and coverage penalties, reduces translation errors by an average of 60% compared to phrase-based systems.

desk verdict GNMT reports a big human-eval win on simple sentences over their phrase-based system, but the gains are not isolated from possible data or compute differences. read the letter →

arxiv 1609.08144 v2 pith:IEDUSXUN submitted 2016-09-26 cs.CL cs.AIcs.LG

Yonghui Wu , Mike Schuster , Zhifeng Chen , Quoc V. Le , Mohammad Norouzi , Wolfgang Macherey , Maxim Krikun , Yuan Cao
show 23 more authors
Qin Gao Klaus Macherey Jeff Klingner Apurva Shah Melvin Johnson Xiaobing Liu {L}ukasz Kaiser Stephan Gouws Yoshikiyo Kato Taku Kudo Hideto Kazawa Keith Stevens George Kurian Nishant Patil Wei Wang Cliff Young Jason Smith Jason Riesa Alex Rudnick Oriol Vinyals Greg Corrado Macduff Hughes Jeffrey Dean
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords neuralmachinetranslationGNMTdeepLSTMwordpiecesattentionmechanismcoveragepenaltyrarewords
checked against Foundation.DAlembert.InevitabilityFoundation.PhiForcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces GNMT as an end-to-end neural approach to machine translation that addresses the computational expense and rare word problems of earlier NMT systems. It details a model using deep LSTMs with eight layers each in encoder and decoder, residual connections, a parallelized attention mechanism, subword wordpieces for vocabulary, low-precision computation for speed, and beam search augmented with length normalization and coverage penalty. Human evaluations on simple sentences demonstrate a 60% average reduction in translation errors relative to the prior phrase-based production system, while matching state-of-the-art on standard benchmarks. This matters because practical deployment requires both high accuracy and fast operation, which previous neural systems struggled to deliver. If the claim holds, it indicates that neural methods can substantially narrow the quality gap to human translators for at least straightforward text.

What carries the argument

GNMT: deep 8-layer LSTM encoder-decoder with attention from decoder bottom to encoder top, wordpiece tokenization, low-precision inference, and coverage-penalized beam search.

What would settle it

Re-training the phrase-based system on the same data volume and hardware as GNMT and re-evaluating both on a diverse set of complex sentences would show whether the 60% error reduction is architecture-specific.

Watch

Extended reading notes

Core claim

Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. The attention mechanism connects the bottom layer of the decoder to the top layer of the encoder to improve parallelism. Wordpieces are used for both input and output to handle rare words. Low-precision arithmetic accelerates inference. Beam search incorporates length-normalization and a coverage penalty to encourage complete translations. On WMT'14 benchmarks GNMT achieves competitive results, and human side-by-side evaluation shows it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Load-bearing premise

The performance improvements are due to the specific model architecture and training choices rather than differences in the amount or quality of training data or available compute resources.

Editorial extensions

If this is right

  • Improves handling of rare words by breaking them into common sub-word units.
  • Accelerates training through better parallelism in the attention mechanism.
  • Speeds up translation inference using low-precision arithmetic.
  • Encourages more complete output sentences via the coverage penalty in beam search.
  • Achieves competitive performance on English-to-French and English-to-German WMT benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Similar architectural choices could be applied to other sequence generation tasks beyond translation.
  • The gains might increase further with larger training corpora, but the paper does not test this directly.
  • Results on isolated simple sentences may not fully predict performance on long, context-dependent or technical texts.
  • Adoption in production could shift the default from phrase-based to neural systems if the error reduction holds across domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper presents GNMT, an 8-layer residual LSTM encoder-decoder NMT system with bottom-decoder-to-top-encoder attention, wordpiece subword segmentation, low-precision inference, and coverage-penalized length-normalized beam search. It reports competitive BLEU scores on the WMT'14 English-to-French and English-to-German benchmarks and claims a 60% average reduction in translation errors versus Google's production phrase-based MT system, measured by human side-by-side ratings on a set of isolated simple sentences.

Significance. If the human-evaluation result holds under controlled conditions, the work is significant for showing that deep NMT can be deployed at production scale and can measurably outperform a mature phrase-based system on the metric that matters most to users. Credit is due for the practical engineering contributions (wordpieces for rare-word handling, residual connections and attention placement for training speed, low-precision arithmetic for inference latency) and for providing a direct, falsifiable human comparison rather than relying solely on automatic metrics.

major comments (2)
  1. [§5] §5 (human side-by-side evaluation): the central claim of a 60% average error reduction is supported only by ratings on 'isolated simple sentences'; the manuscript provides no count of sentences, no description of sentence selection or domain, no inter-rater agreement statistics, and no p-value or confidence interval, making it impossible to judge whether the reported gap is robust or generalizes beyond the evaluated regime.
  2. [§5] §5 (comparison to production PBMT baseline): the 60% error-reduction figure is presented without any statement that training-data volume, parallel-corpus composition, or total optimization effort were held constant between GNMT and the phrase-based production system. Because the production baseline may differ in data scale or tuning regime, the result does not isolate the contribution of the architectural choices (residual LSTMs, attention placement, wordpieces, coverage penalty) that the paper highlights.
minor comments (3)
  1. [Abstract] The abstract states that GNMT 'achieves competitive results' on WMT'14 but omits the actual BLEU numbers; adding the precise scores (and the corresponding state-of-the-art references) would make the summary self-contained.
  2. [Model Architecture] The description of the attention mechanism (bottom decoder layer attending to top encoder layer) would be clearer if accompanied by a small equation or diagram in the model-architecture section.
  3. [§5] Table captions for the WMT results should explicitly note the training data size and whether any external monolingual data were used, to allow direct comparison with contemporaneous systems.

Simulated Author's Rebuttal

2 responses · 1 unresolved

Thank you for the detailed review of our paper on Google's Neural Machine Translation system. We appreciate the positive assessment of the work's significance and will address the concerns raised regarding the human evaluation section to improve the manuscript.

read point-by-point responses

  1. Referee: §5 (human side-by-side evaluation): the central claim of a 60% average error reduction is supported only by ratings on 'isolated simple sentences'; the manuscript provides no count of sentences, no description of sentence selection or domain, no inter-rater agreement statistics, and no p-value or confidence interval, making it impossible to judge whether the reported gap is robust or generalizes beyond the evaluated regime.

    Authors: We acknowledge that §5 provides limited details on the human evaluation protocol. In the revised manuscript, we will add a description of the sentence selection process (isolated simple sentences sampled from production traffic and test sets) and the approximate number of sentences rated. We will also clarify that the evaluation involved multiple professional raters performing side-by-side comparisons. However, inter-rater agreement statistics and formal statistical significance tests were not part of the original evaluation design; we will note this as a limitation of the reported result rather than claiming robustness beyond what the data supports. revision: partial

  2. Referee: §5 (comparison to production PBMT baseline): the 60% error-reduction figure is presented without any statement that training-data volume, parallel-corpus composition, or total optimization effort were held constant between GNMT and the phrase-based production system. Because the production baseline may differ in data scale or tuning regime, the result does not isolate the contribution of the architectural choices (residual LSTMs, attention placement, wordpieces, coverage penalty) that the paper highlights.

    Authors: The comparison is intentionally to the deployed production phrase-based system, which represents the state-of-the-art performance achievable with that paradigm at Google, including all available data and tuning efforts. The goal is to show the practical improvement offered by GNMT over the existing production baseline. We agree that this does not constitute a controlled experiment isolating individual model components. In the revision, we will explicitly state in §5 that the PBMT baseline is the fully optimized production system and that the reported improvement reflects the end-to-end difference rather than the effect of any single architectural decision. revision: yes

standing simulated objections not resolved
  • The lack of inter-rater agreement and p-value statistics for the human evaluation, as these were not computed in the original study and raw data may not be available for re-analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical claims rest on independent benchmarks and human evaluation

full rationale

The paper describes GNMT architecture (8-layer residual LSTMs, bottom-decoder attention, wordpieces, coverage penalty) and reports results on WMT'14 benchmarks plus a 60% error reduction from human side-by-side evaluation on held-out simple sentences versus the production PBMT system. These metrics derive from external test sets and separate raters rather than any equation or fit that defines success in terms of the model's own parameters. No self-definitional loops, fitted-input predictions, or load-bearing self-citations appear in the derivation of the central performance claims; the evaluation is presented as an independent measurement of the described system.

Assumptions & free parameters 2 free parameters · 2 assumptions · 1 invented entities

The central claim depends on empirical training of a large neural network whose capacity and efficiency choices (layer depth, precision, vocabulary construction) are selected by hand rather than derived from first principles.

free parameters (2)
  • encoder and decoder depth = 8
    Set to 8 layers each to increase capacity while preserving training parallelism.
  • wordpiece vocabulary size
    Chosen as a limited set of common sub-word units to balance coverage and efficiency.
assumptions (2)
  • domain assumption LSTM layers with residual connections can model long-range dependencies in translation sequences.
    Foundation for the 8-layer encoder-decoder stack.
  • domain assumption Low-precision arithmetic during inference maintains acceptable translation quality.
    Justification for using reduced precision to speed up decoding.
invented entities (1)
  • wordpieces
    purpose: Sub-word units that allow the model to translate rare words without an open vocabulary.
    New tokenization scheme introduced to solve the rare-word problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." pith.science (2026). https://pith.science/paper/IEDUSXUN

@misc{pith2026160908144,
  author       = {Pith},
  title        = {Pith review of: Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IEDUSXUN}},
  note         = {Machine review of arXiv:1609.08144}
}
read the original abstract

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 184 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 184 Pith citations

  1. Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Data

    cs.LG 2026-03 conditional novelty 8.0 of 10

    ORiGAMi synthesizes sparse semi-structured mixed-type JSON data using path-encoded autoregressive tokenization and schema constraints, outperforming flattened tabular baselines on 17 of 18 fidelity, detection, and uti...

  2. Generative Language Modeling for Automated Theorem Proving

    cs.LG 2020-09 unverdicted novelty 8.0 of 10

    GPT-f, a transformer-based prover for Metamath, generated new short proofs that were accepted into the main library—the first such contribution from a deep-learning system.

  3. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

    cs.CV 2019-08 conditional novelty 8.0 of 10

    A two-stream BERT-style model pretrained on weakly aligned image-caption data transfers to VQA, VCR, referring expressions, and retrieval, outperforming task-specific models on all four.

  4. Verifiably grounded machine interpretation of lunar geology

    cs.CL 2026-08 accept novelty 7.0 of 10

    SelenoVLM generates raster-verifiable lunar geological descriptions, yet its numeric ages default to the global mare prior unless the published age is injected through retrieval.

  5. Consensus Measures for Unstructured Biomedical Text Annotations

    cs.CL 2026-08 conditional novelty 7.0 of 10

    A systematic comparison of soft inter-rater reliability measures for free-text biomedical annotations finds that NLI-based measures are the most robust compromise.

  6. Revisiting Neural Processes via Fourier Transform and Volterra Series

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Set Fourier convolutions plus a Volterra cascade yield scalable, translation-equivariant CNPs that handle irregular inputs with global receptive fields and beat strong baselines.

  7. Tokenization with Split Trees

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    ToaST uses vocabulary-independent split trees and integer programming to produce tokenizers with over 11% fewer tokens than BPE, WordPiece, and UnigramLM while improving 1.5B-parameter LM scores on CORE.

  8. PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    PluRule is a new multimodal multilingual benchmark showing that state-of-the-art vision-language models perform only marginally better than a trivial baseline at detecting specific rule violations in pluralistic onlin...

  9. DaiFu: In-Situ Crash Recovery for Deep Learning Systems

    cs.SE 2025-07 conditional novelty 7.0 of 10

    DaiFu rewrites a DL training function into exception-wrapped cells so a crash can be patched and resumed in place, restoring runs in seconds with under 0.4% overhead.

  10. Tree Transformers are an Ineffective Model of Syntactic Constituency

    cs.CL 2024-11 conditional novelty 7.0 of 10

    Tree Transformers induce constituent structures that diverge from linguistic expectations and provide only marginal gains over standard BERT on hierarchy-sensitive agreement tasks.

  11. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

    cs.LG 2024-02 unverdicted novelty 7.0 of 10

    Griffin hybrid model matches Llama-2 performance while trained on over 6 times fewer tokens and offers lower inference latency with higher throughput.

  12. Massive Activations in Large Language Models

    cs.CL 2024-02 unverdicted novelty 7.0 of 10

    Massive activations are constant large values in LLMs that function as indispensable bias terms and concentrate attention probabilities on specific tokens.

  13. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    cs.CV 2023-10 unverdicted novelty 7.0 of 10

    A new shared video-image tokenizer enables large language models to surpass diffusion models on standard visual generation benchmarks.

  14. iBOT: Image BERT Pre-Training with Online Tokenizer

    cs.CV 2021-11 unverdicted novelty 7.0 of 10

    iBOT achieves 82.3% linear probing accuracy and 87.8% fine-tuning accuracy on ImageNet-1K using masked image modeling with a jointly trained online tokenizer.

  15. Learning to summarize from human feedback

    cs.CL 2020-09 conditional novelty 7.0 of 10

    Reinforcement learning on a reward model trained from human summary comparisons produces summaries humans prefer over supervised fine-tuning or human references on TL;DR and transfers to CNN/DM.

  16. Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity

    cs.CL 2019-09 conditional novelty 7.0 of 10

    LIBERT, a BERT variant pretrained with an auxiliary word-pair similarity task, outperforms BERT on 9/10 GLUE tasks and on three lexical simplification datasets.

  17. On the Downstream Performance of Compressed Word Embeddings

    cs.LG 2019-09 conditional novelty 7.0 of 10

    The eigenspace overlap score, a subspace-preservation measure between compressed and uncompressed embeddings, predicts downstream performance and selects better compressed embeddings more reliably than prior quality measures.

  18. On NMT Search Errors and Model Errors: Cat Got Your Tongue?

    cs.CL 2019-08 conditional novelty 7.0 of 10

    Beam search misses the global best translation for over half of sentences; under exact search, neural MT models prefer the empty translation for more than 50% of inputs.

  19. Text Summarization with Pretrained Encoders

    cs.CL 2019-08 accept novelty 7.0 of 10

    BERTSUM adapts BERT for summarization with per-sentence [CLS] tokens, interval segment embeddings, separate optimizers, and two-stage fine-tuning, reaching state-of-the-art ROUGE scores on CNN/DailyMail, NYT, and XSum.

  20. LXMERT: Learning Cross-Modality Encoder Representations from Transformers

    cs.CL 2019-08 accept novelty 7.0 of 10

    LXMERT pretrains a three-encoder Transformer on image-sentence pairs with five tasks and achieves state-of-the-art VQA, GQA, and NLVR2 results after fine-tuning.

  21. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    cs.CL 2018-08 accept novelty 7.0 of 10

    SentencePiece trains subword models directly from raw text to enable language-independent neural text processing.

  22. Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Tokenization invariance does not generalize beyond English: non-canonical segmentations cut multilingual LLM task scores by ~10–24% on average, worse for high-fragmentation languages, and multi-tokenization LoRA mitig...

  23. LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation

    cs.IR 2026-07 accept novelty 6.5 of 10

    LBR removes length bias in LLM recommenders via length-aware attention offsets and Trie-branching information-length normalization, improving accuracy and fairness with negligible cost.

  24. Jointly Predicting Courses and Grades Using a Transformer-Based Model

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A Transformer that jointly predicts next-semester courses and grades (TRACE) reduces grade MAE by 46% over a grade-only Transformer on a single university's ten-year dataset.

  25. Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Released BPE vocabularies can be used to estimate per-token frequency ratios of a hidden training corpus, with mean relative errors as low as 3% in controlled settings and around 6% on SmolLM.

  26. From token probabilities to calibrated confidence: An empirical study of mathematical question answering

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Sequence-level token-probability averaging yields calibrated confidence for math question answering, and appending the verification prompt to the original generation matches standard re-prompting at 88% lower token-pr...

  27. Mind the Gap: Zero-Query Jailbreaks via Filter-Generator Discrepancy in Text-to-Image Systems

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Zero-query jailbreak attacks on text-to-image systems that exploit filter-generator discrepancy reach 29-33% average success and beat baselines on six pipelines and GPT-image-2.

  28. How Context Attribution Handles What the Model Already Knows

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Context attribution methods cannot disentangle in-context from in-weight knowledge and assign unfaithful scores under overlap; new metrics and WMDP-Cyber++ quantify the failure.

  29. Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies

    cs.LG 2026-07 accept novelty 6.0 of 10

    With a chance/choice distinction, independence of irrelevant alternatives and monotonicity force any choice-rule/value pair to be Boltzmann softmax with a Shannon-entropy bonus and soft Bellman equation.

  30. Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Curating pretraining data for concision in VLMs produces models with up to 35x lower cost-of-pass at matched accuracy by reducing output token count.

  31. OTRO: Oblivious Tokenization Path with Square-Root ORAM

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    OTRO combines replicated square-root ORAM instances, epoch rotation with dummy padding, and KV-cache-aware chunking to make tokenizer lookups oblivious with at most 4.5% TTFT overhead and under 0.5 GB extra memory in ...

  32. Gryphon: A Unified Architecture for Semantic-ID Generation and Item-Level Scoring in Industrial Recommendations

    cs.IR 2026-06 conditional novelty 6.0 of 10

    Gryphon unifies Semantic-ID generation with direct item-level scoring in a single encoder-decoder pass, attaining higher Recall@1000 than vanilla and collision-resolved generative retrieval baselines on an industrial ...

  33. Translators as Invisible Teachers of AI: Copyright, Translation Memory, and the Political Economy of Linguistic Data

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Translators function as unrecognized teachers of AI by supplying translation memories and parallel corpora that power machine translation and LLMs, yet their contributions are processed as information analysis data un...

  34. Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering Design

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    A hybrid agentic architecture integrates knowledge-based physical verification tools into LLM-driven CAD design loops, producing more complex and functionally valid designs than prior agentic baselines.

  35. Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Models multi-head transformer data flow as time-dependent Wasserstein gradient flows of an attention-capturing interaction energy, with proofs on omega-limit stationary points and stability under weight and input pert...

  36. Decaf: Improving Neural Decompilation with Automatic Feedback and Search

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    Decaf uses compiler feedback and search to improve neural decompilation, boosting semantic success rate from 26.0% to 83.9% on ExeBench Real -O2 split.

  37. SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    SimCT recovers discarded teacher signal in cross-tokenizer on-policy distillation by enlarging supervision to jointly realizable multi-token continuations, yielding consistent gains on math reasoning and code generati...

  38. MASPRM: Multi-Agent System Process Reward Model

    cs.MA 2025-10 conditional novelty 6.0 of 10

    MASPRM trains a per-agent, per-message value head from outcome-only MCTS rollouts and uses it to guide step-level beam search and MCTS, improving exact match on GSM8K by up to +30.7 points over a greedy multi-agent pass.

  39. Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A word-as-image pixel language model trained with next-token prediction reports lower perplexity than a token-embedding LLaMA on noisy and non-Latin-script text, though its noise evaluation holds tokenization fixed.

  40. FlowletFormer: Network Behavioral Semantic Aware Pre-training Model for Traffic Classification

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlowletFormer is a BERT-style pre-training model for network traffic that uses flowlet segmentation, field-level tokenization, and two self-supervised objectives, reaching state-of-the-art classification on 7 of 8 pub...

  41. When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Projecting away the estimated Modern Standard Arabic subspace during fine-tuning improves generation across 25 Arabic dialects by up to +4.9 chrF++, evidence that subspace dominance by a high-resource variety restrict...

  42. Anomaly Detection for IoT Global Connectivity

    cs.NI 2025-08 conditional novelty 6.0 of 10

    An unsupervised roaming-signaling pipeline flags IoT fleet connectivity issues, but the headline results are weakened by training/test overlap.

  43. Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice

    cs.CL 2025-07 conditional novelty 6.0 of 10

    An end-to-end simultaneous speech-to-speech translation model with voice cloning, trained with a two-stage reinforcement learning reward scheme, reports high accuracy and low latency on the authors' RealSI benchmark.

  44. Step-Audio 2 Technical Report

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    Step-Audio 2 integrates a latent audio encoder, reasoning-centric reinforcement learning, and discrete audio token generation into language modeling to deliver state-of-the-art performance on audio understanding and c...

  45. Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis

    cs.SE 2025-06 conditional novelty 6.0 of 10

    An empirical study of 38,742 issue reports and 19 interviews produces a 20-theme, 75-sub-theme taxonomy of LLM-centric framework challenges and five recommendations.

  46. Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A Lyapunov-spectrum-based distance to the dense network lets hyperparameter search for pruned RNNs stop early and select models that beat both loss-based baselines and the dense originals.

  47. Incorporating Domain Knowledge into Materials Tokenization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A domain-knowledge-weighted tokenizer, MATTER, preserves material concepts and improves materials NLP performance by 4% on generation and 2% on classification tasks.

  48. TokAlign: Efficient Vocabulary Adaptation via Token Alignment

    cs.CL 2025-06 conditional novelty 6.0 of 10

    TokAlign aligns source and target BPE token vocabularies using GloVe co-occurrence embeddings and re-initializes LLM embeddings, recovering within 5k steps and enabling token-level distillation.

  49. Who Reasons in the Large Language Models?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    The authors hypothesize, with evidence from merging and freezing experiments, that the attention output projection is the primary locus of mathematical reasoning in large language models.

  50. Hallucinations are inevitable but can be made statistically negligible

    cs.CL 2025-02 unverdicted novelty 6.0 of 10

    Hallucinations are inevitable on an infinite set of inputs but can be made statistically negligible with sufficient training data quality and quantity.

  51. Theoretical Benefit and Limitation of Diffusion Language Model

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.

  52. Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo

    cs.LG 2025-02 conditional novelty 6.0 of 10

    An SMC importance-sampling algorithm debiases discrete diffusion guidance, asymptotically sampling from the target tempered distribution p0(x0)p(ζ|x0)^α.

  53. Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers

    cs.SD 2025-02 conditional novelty 6.0 of 10

    A transformer speech encoder can learn to internally rearrange audio information into text order, enabling a lightweight decoder trained with simple cross-entropy to nearly match RNN-Transducer accuracy with faster inference.

  54. Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sparse, 8-bit quantized S5 linear RNNs match dense model audio denoising accuracy with 2x less compute and 36% less memory, and run 42x faster with 149x lower energy on Loihi 2 than a dense FP32 model on Jetson Orin Nano.

  55. Multiband Embeddings of Light Curves

    astro-ph.IM 2025-01 conditional novelty 6.0 of 10

    A modular ensemble of per-band LSTMs with a central merging LSTM classifies multiband light curves without interpolation, matching or beating feature-based random forests especially when observations are sparse.

  56. ORIGAMI: A generative transformer architecture for predictions from semi-structured data

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ORIGAMI is a generative transformer with structure-preserving tokenization, key/value position encoding, and grammar-constrained decoding that matches or beats baselines on tabular, multi-label, and code-classification tasks.

  57. Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Output token count, observable through response timing, can reveal a user's target language or classification result with 70-87% accuracy in the authors' experiments.

  58. HARP: Hesitation-Aware Reframing in Transformer Inference Pass

    cs.CL 2024-12 conditional novelty 6.0 of 10

    HARP improves LLM accuracy by up to 5.16% at about 1.25x inference cost, by running a dropout-perturbed second forward pass only when token-level entropy is high.

  59. MVD: A Multi-Lingual Software Vulnerability Detection Framework

    cs.SE 2024-12 reject novelty 6.0 of 10

    MVD trains a multi-lingual CodeBERT classifier with a focal-plus-logit-adjustment loss and distillation-based incremental learning, claiming large gains over single-language baselines that mostly vanish under fair comparison.

  60. ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain

    cs.CL 2024-11 conditional novelty 6.0 of 10

    ChemTEB is a new open-source benchmark of 35 chemistry text tasks, and on it OpenAI text-embedding-3-large ranks first overall, Nomic Embedding v1.5 is the best open-source model, and SMILES-to-text matching is the ha...

See all 184 Pith citations

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · cited by 184 Pith papers (see all)

  1. [1]

    G., Steiner, B., Tucker, P., V asudevan, V., W arden, P., Wicke, M., Yu, Y., and Zheng, X

    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., V asudevan, V., W arden, P., Wicke, M., Yu, Y., and Zheng, X. Tensorflow: A system for large-scale machine learning. Tech. rep., Google Brain, 2016. arXiv preprint

    work page 2016

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. InInternational Conference on Learning Representations(2015)

    work page 2015

  3. [3]

    D., Pietra, V

    Brown, P., Cocke, J., Pietra, S. D., Pietra, V. D., Jelinek, F., Mercer, R., and Roossin, P. A statistical approach to language translation. InProceedings of the 12th Conference on Computational Linguistics - Volume 1(Stroudsburg, PA, USA, 1988), COLING ’88, Association for Computational Linguistics, pp. 71–76

    work page 1988

  4. [4]

    F., Cocke, J., Pietra, S

    Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J. D., Mercer, R. L., and Roossin, P. S. A statistical approach to machine translation.Computational linguistics 16, 2 (1990), 79–85

    work page 1990

  5. [5]

    F., Pietra, V

    Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. The mathematics of statistical machine translation: Parameter estimation.Comput. Linguist. 19, 2 (June 1993), 263–311

    work page 1993

  6. [6]

    N-gram counts and language models from the common crawl

    Buck, C., Heafield, K., and V an Ooyen, B. N-gram counts and language models from the common crawl. InLREC (2014), vol. 2, Citeseer, p. 4

    work page 2014

  7. [7]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation

    Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing(2014)

    work page 2014

  8. [8]

    Learning recursive distributed representations for holistic computation.Connection Science 3, 4 (1991), 345–366

    Chrisman, L. Learning recursive distributed representations for holistic computation.Connection Science 3, 4 (1991), 345–366. 20

    work page 1991

Show all 44 references
  1. [10]

    A character-level decoder without explicit segmentation for neural machine translation.CoRR abs/1603.06147 (2016)

    Chung, J., Cho, K., and Bengio, Y. A character-level decoder without explicit segmentation for neural machine translation.CoRR abs/1603.06147 (2016)

    2016

  2. [11]

    R., and Fonollosa, J

    Costa-Jussà, M. R., and Fonollosa, J. A. R. Character-based neural machine translation.CoRR abs/1603.00810 (2016)

    2016

  3. [12]

    S., Monga, R., Chen, K., Devin, M., Le, Q

    Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. Large scale distributed deep networks. In NIPS (2012)

    2012

  4. [13]

    M., and Makhoul, J

    Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R. M., and Makhoul, J. Fast and robust neural network joint models for statistical machine translation. InACL (1)(2014), Citeseer, pp. 1370–1380

    2014

  5. [14]

    Multi-task learning for multiple language translation

    Dong, D., Wu, H., He, W., Yu, D., and W ang, H. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (2015), pp. 1723–1732

    2015

  6. [15]

    Edinburgh’s phrase-based machine translation systems for WMT-14

    Durrani, N., Haddow, B., Koehn, P., and Heafield, K. Edinburgh’s phrase-based machine translation systems for WMT-14. InProceedings of the Ninth Workshop on Statistical Machine Translation (2014), Association for Computational Linguistics Baltimore, MD, USA, pp. 97–104

    2014

  7. [16]

    E., and Lebiere, C

    F ahlman, S. E., and Lebiere, C. The cascade-correlation learning architecture. InAdvances in Neural Information Processing Systems 2(1990), Morgan Kaufmann, pp. 524–532

    1990

  8. [17]

    A., Schmidhuber, J., and Cummins, F

    Gers, F. A., Schmidhuber, J., and Cummins, F. Learning to forget: Continual prediction with LSTM. Neural computation 12, 10 (2000), 2451–2471

    2000

  9. [18]

    Pointing the unknown words

    Gülçehre, Ç., Ahn, S., Nallapati, R., Zhou, B., and Bengio, Y. Pointing the unknown words. CoRR abs/1603.08148 (2016)

    2016

  10. [19]

    Deep learning with limited numerical precision.CoRR abs/1502.02551 (2015)

    Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision.CoRR abs/1502.02551 (2015)

    2015

  11. [20]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.CoRR abs/1510.00149 (2015)

    2015

  12. [21]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition(2015)

    2015

  13. [22]

    Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001

    Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001

    2001

  14. [23]

    Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

    Hochreiter, S., and Schmidhuber, J. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

    1997

  15. [24]

    Recurrent continuous translation models

    Kalchbrenner, N., and Blunsom, P. Recurrent continuous translation models. InConference on Empirical Methods in Natural Language Processing(2013)

    2013

  16. [25]

    P., and Ba, J

    Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization.CoRR abs/1412.6980 (2014)

    2014 arXiv

  17. [26]

    J., and Marcu, D

    Koehn, P., Och, F. J., and Marcu, D. Statistical phrase-based translation. InProceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics(2003)

    2003

  18. [27]

    Ternary weight networks.CoRR abs/1605.04711 (2016)

    Li, F., and Liu, B. Ternary weight networks.CoRR abs/1605.04711 (2016). 21

    2016

  19. [28]

    Luong, M., and Manning, C. D. Achieving open vocabulary neural machine translation with hybrid word-character models.CoRR abs/1604.00788 (2016)

    2016

  20. [29]

    V., Sutskever, I., Vinyals, O., and Kaiser, L

    Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., and Kaiser, L. Multi-task sequence to sequence learning. InInternational Conference on Learning Representations(2015)

    2015

  21. [30]

    Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. InConference on Empirical Methods in Natural Language Processing(2015)

    2015

  22. [31]

    V., Vinyals, O., and Zaremba, W

    Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., and Zaremba, W. Addressing the rare word problem in neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural ...

    2015

  23. [32]

    Reward augmented maximum likelihood for neural structured prediction

    Norouzi, M., Bengio, S., Chen, Z., Jaitly, N., Schuster, M., Wu, Y., and Schuurmans, D. Reward augmented maximum likelihood for neural structured prediction. InNeural Information Processing Systems(2016)

    2016

  24. [33]

    Understanding the exploding gradient problem.CoRR abs/1211.5063 (2012)

    Pascanu, R., Mikolov, T., and Bengio, Y. Understanding the exploding gradient problem.CoRR abs/1211.5063 (2012)

    2012

  25. [34]

    Sequence level training with recurrent neural networks

    Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. InInternational Conference on Learning Representations(2015)

    2015

  26. [35]

    Japanese and Korean voice search.2012 IEEE International Conference on Acoustics, Speech and Signal Processing(2012)

    Schuster, M., and Nakajima, K. Japanese and Korean voice search.2012 IEEE International Conference on Acoustics, Speech and Signal Processing(2012)

    2012

  27. [36]

    Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing 45, 11 (Nov

    Schuster, M., and Paliwal, K. Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing 45, 11 (Nov. 1997), 2673–2681

    1997

  28. [37]

    On using very large target vocabulary for neural machine translation

    Sébastien, J., Kyunghyun, C., Memisevic, R., and Bengio, Y. On using very large target vocabulary for neural machine translation. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan...

    2015

  29. [38]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    2016

  30. [39]

    Minimum risk training for neural machine translation

    Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., and Liu, Y. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    2016

  31. [40]

    K., Greff, K., and Schmidhuber, J

    Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks.CoRR abs/1505.00387 (2015)

    2015

  32. [41]

    Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems(2014), pp. 3104–3112

    2014

  33. [42]

    Coverage-based neural machine translation

    Tu, Z., Lu, Z., Liu, Y., Liu, X., and Li, H. Coverage-based neural machine translation. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics(2016)

    2016

  34. [43]

    Quantized convolutional neural networks for mobile devices

    Wu, J., Leng, C., W ang, Y., Hu, Q., and Cheng, J. Quantized convolutional neural networks for mobile devices. CoRR abs/1512.06473 (2015)

    2015

  35. [44]

    Recurrent neural network regularization, 2014

    Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization, 2014

    2014

  36. [45]

    NMT before RL

    Zhou, J., Cao, Y., W ang, X., Li, P., and Xu, W. Deep recurrent models with fast-forward connections for neural machine translation.CoRR abs/1606.04199 (2016). 22 Table 11: Some example translations from PBMT [15], our GNMT system (the "NMT before RL", Table 9), and Human. Sou...

    2016

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.