REVIEW 63 cited by
FastText.zip: Compressing text classification models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent quantization artefacts. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.
Forward citations
Showing 60 of 63 Pith papers that cite this
-
From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills
Over 99% of real-world SKILL.md files contain skill smells (violations of authoring best practices), and those smells almost never disappear as the skills evolve.
-
Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Gemma 3 27B and Aya Expanse 32B are the strongest multilingual synthetic-data teachers; model scale does not predict effectiveness while prompt diversity, length and response fluency do.
-
Euska\~nolDS: A Naturally Sourced Corpus for Basque-Spanish Code-Switching
A new corpus of 20,008 naturally sourced Basque-Spanish code-switched sentences, including a 927-sentence manually verified gold subset, with a qualitative typology of the switches.
-
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
A new Common Crawl-derived dataset, built by ensembling quality classifiers and adding 1.9T synthetic rewrites, lets an 8B model trained for 15T tokens beat Llama 3.1 8B on MMLU and several other tasks.
-
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
Hard negatives selected by visual confusability in sign embeddings, not linguistic similarity, substantially raise fine-grained sign-language retrieval accuracy without collapsing coarse performance.
-
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.
-
Pre-Training on Software Engineering Texts: Effects on Domain Adaptation and General-Language Understanding
Under constant-token and compute-matched budgets, continual pre-training on SE text slightly improves domain scores while preserving general language ability, whereas pre-training from scratch incurs large decisive lo...
-
MORE: A Multilingual Document Parsing Benchmark and Evaluation
MORE provides a 149-language, structure-aware document parsing benchmark from real PDFs and reports baselines showing specialized OCR models still fail on tables and rare scripts.
-
On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
Systematic experiments reveal that activation steering trades fluency for concept control, is less effective on instruction-tuned models, and that prompting/SFT excel at injection but not removal, with textual metrics...
-
Language-Aware Token Boosting: LLM Language Confusion Reduction Without Tuning
LATB reduces LLM language confusion via language-specific token probability boosts at inference without model tuning while preserving summarization quality.
-
Steering Vectors are an Adversarial Attack Surface
Poisoning 4-6% of tokens in activation steering datasets produces vectors that jailbreak LLMs with 20-55% attack success rate while preserving benign steering effects.
-
Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
Multilingual SAEs strengthen cross-lingual representations for reliable steering and an intersection-based rule selects effective layers without exhaustive search.
-
TeraGram: A Structured Longitudinal Dataset of the Telegram Messenger
A large-scale longitudinal dataset of public Telegram content is introduced to enable studies of engagement patterns and network evolution without algorithmic curation.
-
DocAtlas: Multilingual Document Understanding Across 80+ Languages
DocAtlas introduces model-free rendering pipelines to create DocTag-annotated datasets across 82 languages and shows DPO adaptation improves multilingual performance without base-language degradation.
-
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
CommonLID, a human-annotated 109-language web-text benchmark, shows current language-identification models score roughly 60-70% F1 on noisy web data, well below their clean-test-set numbers.
-
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Classifier-based quality filtering for LLM pretraining improves downstream tasks by implicitly filtering the reference high-quality set rather than by mimicking it, and its quality scores fail a data-conditioning test.
-
NeoBabel: A Multilingual Open Tower for Visual Generation
A 2B multilingual text-to-image model trained on 124M translated pairs matches or beats larger English-only baselines on English while scoring higher on the authors' multilingual benchmark extensions.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
Magistral
Pure RL alone, without distilled reasoning traces, turned Mistral's base models into strong reasoning models on math and coding benchmarks.
-
Seed-Coder: Let the Code Model Curate Data for Itself
Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.
-
StRuCom: A Novel Dataset of Structured Code Comments in Russian
StRuCom is the first large-scale Russian dataset of structured code comments, built from GitHub and LLM-generated data, and fine-tuning code models on it improves Russian docstring generation quality.
-
IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation
IRLBench provides 1,700 parallel English-Irish exam questions and shows state-of-the-art LLMs perform markedly worse in Irish than in English.
-
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
AttentionInfluence uses retrieval-head masking loss differences in a 1.3B model to select 73B tokens that improve a 7B model's reasoning benchmarks by 1.4 to 3.5pp.
-
Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data
Ultra-FineWeb is a fastText-filtered pretraining corpus whose seed samples were chosen by a cheap 'efficient verification' step, and 1.2B models trained on it outperform models trained on FineWeb and FineWeb-edu on av...
-
Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
CLIMB automatically discovers pre-training data mixtures by clustering text embeddings and iteratively refining mixture weights with a predictor, improving 1B-model reasoning accuracy over standard baselines.
-
Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
Preference examples vary in difficulty; overly difficult examples degrade DPO alignment, and filtering them out improves AlpacaEval 2 win rates by 9-16 percentage points.
-
From Thought to Action: How a Hierarchy of Neural Dynamics Supports Language Production
During typing, the brain sequentially represents sentence context, then words, syllables, and letters, and these representations overlap in time and are carried by neural codes that change faster for lower-level features.
-
Enhancing Web Service Anomaly Detection via Fine-grained Multi-modal Association and Frequency Domain Analysis
FFAD detects web service anomalies by building a complete log-metric graph, applying a Fourier graph operator, and down-weighting high-energy frequency components during reconstruction.
-
From Division to Unity: A Large-Scale Study on the Emergence of Computational Social Science, 1990-2021
Computational social science grew in two waves, formed a distinct identity between 2000 and 2014, then diffused back into sociology, politics, psychology, and economics.
-
LogLLM: Log-based Anomaly Detection Using Large Language Models
LogLLM combines BERT message embeddings with Llama sequence classification via a learned projector and regex preprocessing, reporting higher F1 scores than nine baselines on four log datasets.
-
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
An open-source MoE code model matches GPT-4 Turbo on coding and math benchmarks while expanding to 338 languages and 128K context length.
-
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
DeepSeekMath 7B reaches 51.7% on MATH via continued pretraining on curated web math data and Group Relative Policy Optimization.
-
Studying Lobby Influence in the European Parliament
NLP comparison of lobby papers and MEP speeches discovers influence links validated indirectly via retweets and meetings, achieving AUC 0.77 and ideological alignment in aggregate analysis.
-
Few-shot Text Classification with Distributional Signatures
A few-shot text classifier that learns attention from distributional word statistics (frequency and class skew) generalizes to unseen classes better than lexical-feature meta-learners.
-
AIBench: An Industry Standard Internet Service AI Benchmark Suite
AIBench is a modular benchmark suite, with sixteen AI component benchmarks and an end-to-end e-commerce search workload, together with preliminary CPU and GPU characterization results.
-
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
Gated fusion of fastText and BERT embeddings into an end-to-end ASR model captures multi-sentence conversational context and lowers word error rate on the Switchboard corpus.
-
Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry
CLR-transformed unigram+bigram profiles classified by Aitchison distance reach 84%/95.6%/100% accuracy on short/medium/long texts across six Latin-alphabet languages.
-
Structural Under-Representation of Women in News: Nonparametric Bayesian Mixtures Capture Time-Dependent Dynamics
Nonparametric Bayesian mixtures on Canadian news data reveal structural under-representation of women as sources, driven more by topic than region, with over 85% of series showing no improvement toward parity and stab...
-
Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education
Compares LLMs against semantic similarity for binary classification of student self-explanations in programming education.
-
Weakly Supervised Medical Entity Extraction and Linking for Chief Complaints
A split-and-match weak supervision pipeline trains BERT and BiLSTM models to extract and link medical entities from chief complaints without human annotation, achieving 67.5 F1 on a clinician-labeled test set.
-
Belief Alignment vs Opinion Leadership: Understanding Cross-linguistic Digital Activism in K-pop and BLM Communities
Using 29 million tweets, the study concludes that K-pop fans joined BLM activism mainly because they already shared the movement's beliefs, not because BTS led them, though the evidence is only moderate.
-
Iterative Augmentation with Summarization Refinement (IASR) Evaluation for Unstructured Survey data Modeling and Analysis
The paper evaluates four LLMs as text augmenters and reports GPT-3.5 Turbo as the best, and that combining augmentation with GPT topic labels increases BERTopic's discovered topics from 5 to 20 with zero overlap.
-
Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models
Adding parallel data during continued pretraining improves a 1.1B LLM's translation and multilingual common-sense reasoning, with end-of-training placement performing best.
-
dots.llm1 Technical Report
A 14B-active MoE model roughly matches Qwen2.5-72B on a broad benchmark suite while reporting about a 4x reduction in training GPU-hours.
-
OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning
OWSM v4 models, trained on a cleaned 166k-hour multilingual YODAS subset, beat prior open OWSM models and are competitive with Whisper and MMS on several benchmarks.
-
Semantic Pivots Enable Cross-Lingual Transfer in Large Language Models
LLMs often translate words through intermediate 'semantic pivot' tokens, and pretraining data enriched with such pivots modestly improves cross-lingual word translation.
-
Think Only When You Need with Large Hybrid-Reasoning Models
The authors introduce a two-stage training pipeline (supervised fine-tuning plus reinforcement learning) that lets a language model adaptively choose between a reasoning mode and a direct-answer mode, and a new metric...
-
PREMISE: Matching-based Prediction for Accurate Review Recommendation
PREMISE, a matching-score-based model, achieves state-of-the-art results on multimodal review helpfulness prediction on Amazon-MRHP and Lazada-MRHP, improving MAP by over 5 points versus the strongest baseline.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
-
Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation
EcoDatum filters web image-text data by ensembling eight unimodal and multimodal quality scorers with weak-supervision weighting, reporting a DataComp small-scale average score of 0.182.
-
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study
A new data-mixing recipe (Parallel-First Monolingual-Second) and a 9B model, GemmaX2-28, achieve translation quality competitive with Google Translate and GPT-4 across 28 languages.
-
RedStone: Curating General, Code, Math, and QA Data for Large Language Models
RedStone turns Common Crawl into large, diverse pre-training datasets for language, code, math, and QA, claiming improved benchmark scores over existing open datasets.
-
Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script
A Tibetan syllable-level black-box attack using syllable embeddings and a probability-based scoring mechanism successfully fools fine-tuned CINO models, with attack success rates up to 76%.
-
MOSS-Audio Technical Report
MOSS-Audio is an audio-language model using a 12.5 Hz encoder, DeepStack cross-layer injection, time markers, and an event-preserving annotation pipeline for unified audio understanding.
-
Toward LLMs Beyond English-Centric Development
Analysis of open-weight LLMs reveals strong English bias in generated sequences, with continual pre-training providing no cost benefit over from-scratch training for non-English adaptation.
-
Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs
An embedding-drift prompt injection detector that is not zero-shot, requires a clean reference prompt at inference, and fits its threshold on the test set, so the reported >93% accuracy is not evidence of deployed per...
-
Boosting Data Utilization for Multilingual Dense Retrieval
A three-stage data-utilization pipeline for multilingual dense retrieval, combining ensemble hard-negative mining, LLM-based filtering/generation, and monolingual topic-diverse mini-batches, improves MIRACL nDCG@10 by...
-
A Combined Feature Embedding Tools for Multi-Class Software Defect and Identification
CodeGraphNet, a GraphCodeBERT-plus-GCN embedding with a DeepTree classifier, is claimed to detect five CWE vulnerability classes at 98% accuracy, but its own unseen-data results are much lower (76-87%) and baseline co...
-
Context-aware Deep Model for Entity Recommendation in Search Engine at Alibaba
A BiLSTM-plus-attention model learns query and entity embeddings jointly from search logs and recommends entities for arbitrary Chinese search queries without requiring an explicit entity in the query.
-
PortBERT: Navigating the Depths of Portuguese Language Models
PortBERT releases two RoBERTa models for Portuguese that match or beat prior monolingual and multilingual models on translated GLUE/SuperGLUE tasks while reporting training and inference times.
Discussion (0). Continue with ORCID to comment.