Pith. sign in

REVIEW 5 cited by

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.12035 v2 pith:7AQOW4YK submitted 2022-05-24 cs.CL

Shitao Xiao , Zheng Liu , Yingxia Shao , Zhao Cao This is my paper
classification cs.CL
keywords maskeddecoderencoderpre-trainingretromaesentenceinputmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite pre-training's progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval. In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE). RetroMAE is highlighted by three critical designs. 1) A novel MAE workflow, where the input sentence is polluted for encoder and decoder with different masks. The sentence embedding is generated from the encoder's masked input; then, the original sentence is recovered based on the sentence embedding and the decoder's masked input via masked language modeling. 2) Asymmetric model structure, with a full-scale BERT like transformer as encoder, and a one-layer transformer as decoder. 3) Asymmetric masking ratios, with a moderate ratio for encoder: 15~30%, and an aggressive ratio for decoder: 50~70%. Our framework is simple to realize and empirically competitive: the pre-trained models dramatically improve the SOTA performances on a wide range of dense retrieval benchmarks, like BEIR and MS MARCO. The source code and pre-trained models are made publicly available at https://github.com/staoxiao/RetroMAE so as to inspire more interesting research.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mutual Linearity in and out of Stationarity for Markov Jump Processes: A Trajectory-Based Approach

    cond-mat.stat-mech 2026-04 unverdicted novelty 7.0 of 10

    Trajectory-level linear response yields mutual linearity of observables under single-edge rate perturbation for Markov jump processes, including non-stationary state and counting observables.

  2. Separating Representation from Reconstruction Enables Scalable Text Encoders

    cs.CL 2026-07 accept novelty 6.5 of 10

    Separating representation from token reconstruction via a bipartite CrossBERT architecture restores scalable frozen text embeddings and enables high-masking complementary training.

  3. From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection

    cs.CR 2025-07 conditional novelty 6.0 of 10

    SHIELD, an LLM-aided pipeline combining a masked autoencoder, deterministic data augmentation, and multi-level prompting, detects host-based attacks with high precision on three public datasets.

  4. HyReC: Exploring Hybrid-based Retriever for Chinese

    cs.IR 2025-06 conditional novelty 6.0 of 10

    HyReC unifies dense, lexicon, and learned word-segment retrieval into one model and reports improved C-MTEB retrieval scores for Chinese.

  5. A Comparative Study of Specialized LLMs as Dense Retrievers

    cs.IR 2025-07 conditional novelty 5.0 of 10

    Specialized Qwen2.5 7B models differ in dense retrieval quality: math and long-reasoning variants degrade performance, while coder and vision-language variants improve zero-shot text and code retrieval.

Pith tools