REVIEW 18 cited by
MPNet: Masked and Permuted Pre-training for Language Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
MPNet: Masked and Permuted Pre-training for Language Understanding
read the original abstract
BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet.
Forward citations
Cited by 18 Pith papers
-
Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks
The paper proves multi-layer Transformer attention converges to Hasse diagrams and gives a constructive method to derive attention masks via minimal common supergraphs of partial orders.
-
A Woman with a Knife or A Knife with a Woman? Measuring Directional Bias Amplification in Image Captions
DBAC is a new directional metric for bias amplification in image captions that is less sensitive to sentence encoders and more accurate than LIC, validated on COCO gender and race attributes.
-
DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing
A production LLM-based system validates 1,013 mobile flows on every code change, reporting 91.6% iOS and 92.2% Android pass rates while cutting test onboarding from ~4 days to under 4 hours.
-
CoPersona: Collaborative Persona Graphs for Robust LLM Personalization
CoPersona introduces a multiplex persona graph for facet-level peer alignment and a dual-branch retrieval-plus-reasoning architecture to improve LLM personalization under sparse and biased user interaction data.
-
ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification
ConvMemory v3 introduces a dual-evidence gate for target-conditioned memory validity verification, reporting 90.12% accuracy on synthetic benchmarks, 98.8% transfer to real data, and H@1 improvement from 45.1% to 95.7...
-
Story Operators: Decomposing the Original $\to$ Sequel Transformation in Embedding Space
Sequels in embedding space decompose into low-rank interpretable axes, yielding a taxonomy of formulaic, concentrated, and compositional types, with Tom Sawyer to Huckleberry Finn dominated by a structural domestic-to...
-
The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
LLM routers across 21 methods on 5 benchmarks converge to similar accuracy below oracle due to learning global performance trends rather than fine-grained query signals.
-
XSearch: Explainable Code Search via Concept-to-Code Alignment
XSearch achieves 15x gains on out-of-distribution code search benchmarks by replacing global embedding similarity with explicit concept-to-statement alignment.
-
XSearch: Explainable Code Search via Concept-to-Code Alignment
XSearch achieves explainable code search by breaking queries into functional concepts and matching them directly to code statements, delivering large gains on out-of-distribution benchmarks.
-
Finding Meaning in Embeddings: Concept Separation Curves
Concept Separation Curves provide a classifier-independent method to visualize and quantify how sentence embeddings distinguish conceptual meaning from syntactic variations across languages and domains.
-
TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings
TriagerX combines dual-transformer content rankings with developer interaction history to improve top-k accuracy for developer and component recommendations in bug triaging across five datasets.
-
SemStruct: Contextualizing Semantic Embeddings with Structural Information for Schema Matching
SemStruct models tables as heterogeneous graphs with GNNs on frozen PLM embeddings to incorporate row co-occurrences for schema matching and reports SOTA results on Valentine and SOTAB-SM benchmarks.
-
ConvMemory: A Lightweight Learned Memory Reranker, a Negative Attribution Result, and a Research-Preview Conflict Editor
ConvMemory delivers competitive recall at far lower latency than larger rerankers for long-term conversational memory while a multi-seed ablation refutes temporal-structure exploitation as the operative mechanism.
-
VulLink: A Dynamic Open-Access Vulnerability Graph Database for Cybersecurity Data Mining
VulLink continuously ETL-integrates public vulnerability repositories into an open, typed graph with API access and pretrained embeddings for cybersecurity data mining.
-
Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison
Headache specialists preferred their own literature summaries over those from Sonnet, GPT-4o, and Llama 3.1 in a blinded evaluation, though AI summaries were sometimes indistinguishable.
-
To Select or not to Select, that is the Question: Distilling Robot Skill Prediction into a Small Ensemble
A 133M-parameter ensemble of fine-tuned mpnet and MiniLM encoders achieves 83.5% accuracy on a 200-task synthetic benchmark for robot skill prediction, beating several larger zero-shot LLMs.
-
ConvMemory v2: A Recall-Preserving Top-10 Evidence Reranker for Conversational Memory Retrieval
ConvMemory v2 fine-tunes a 22M-parameter MiniLM cross-encoder on v1's top-10 to raise FULL MRR from 0.5824 to 0.6560 and H@1 from 0.4440 to 0.5474 on LoCoMo while preserving Recall@10.
-
VulLink: A Dynamic Open-Access Vulnerability Graph Database for Cybersecurity Data Mining
VulGD is a dynamic open-access graph database that aggregates vulnerability data from multiple sources and uses LLM embeddings to enable more accurate risk assessment and threat prioritization.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.