{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2021:WYOEE2ZBWHIBD7DAZTWK74NMIU","short_pith_number":"pith:WYOEE2ZB","schema_version":"1.0","canonical_sha256":"b61c426b21b1d011fc60ccecaff1ac453976a93258d1f258891638c6bc541e33","source":{"kind":"arxiv","id":"2106.08254","version":2},"attestation_state":"computed","paper":{"title":"BEiT: BERT Pre-Training of Image Transformers","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"BEiT pre-trains vision transformers by recovering discrete visual tokens from masked image patches, reaching 83.2% ImageNet-1K accuracy.","cross_cats":["cs.LG"],"primary_cat":"cs.CV","authors_text":"Furu Wei, Hangbo Bao, Li Dong, Songhao Piao","submitted_at":"2021-06-15T16:02:37Z","abstract_excerpt":"We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first \"tokenize\" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2106.08254","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CV","submitted_at":"2021-06-15T16:02:37Z","cross_cats_sorted":["cs.LG"],"title_canon_sha256":"3b15a271f12d9e3e7cee88cd64608d27a190306c89c598295a2bb6c1c3b42777","abstract_canon_sha256":"1f3e8325d4af081ae4fa89ef02f1e4adc9ee86873be729eb2882fb5efb7f5e21"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T04:54:14.716647Z","signature_b64":"bTRpFhjCDh6amimL6EwslGTfm2TQ4AZnFZ9u8n9qkm68NAMoitFqD2qye+WSSG2eW8yv7mHftKkCfMQVpvgHDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"b61c426b21b1d011fc60ccecaff1ac453976a93258d1f258891638c6bc541e33","last_reissued_at":"2026-07-05T04:54:14.716257Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T04:54:14.716257Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"BEiT: BERT Pre-Training of Image Transformers","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"BEiT pre-trains vision transformers by recovering discrete visual tokens from masked image patches, reaching 83.2% ImageNet-1K accuracy.","cross_cats":["cs.LG"],"primary_cat":"cs.CV","authors_text":"Furu Wei, Hangbo Bao, Li Dong, Songhao Piao","submitted_at":"2021-06-15T16:02:37Z","abstract_excerpt":"We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first \"tokenize\" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The discrete visual tokens produced by the separate tokenizer are assumed to form a sufficiently rich and stable target for the masked modeling objective; if the tokenizer collapses or captures only low-level statistics the pre-training signal would be weak.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"BEiT pre-trains vision transformers via masked image modeling on visual tokens and reaches 83.2% ImageNet top-1 accuracy for the base model and 86.3% for the large model using only ImageNet-1K data.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"BEiT pre-trains vision transformers by recovering discrete visual tokens from masked image patches, reaching 83.2% ImageNet-1K accuracy.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"e39c160f636a9593041cc9e091add1dbce3b713cedea8aa43b9437287a31d40b"},"source":{"id":"2106.08254","kind":"arxiv","version":2},"verdict":{"id":"f0441075-22f4-4727-88b0-a3b4af7ef257","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-13T11:44:06.846514Z","strongest_claim":"base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).","one_line_summary":"BEiT pre-trains vision transformers via masked image modeling on visual tokens and reaches 83.2% ImageNet top-1 accuracy for the base model and 86.3% for the large model using only ImageNet-1K data.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The discrete visual tokens produced by the separate tokenizer are assumed to form a sufficiently rich and stable target for the masked modeling objective; if the tokenizer collapses or captures only low-level statistics the pre-training signal would be weak.","pith_extraction_headline":"BEiT pre-trains vision transformers by recovering discrete visual tokens from masked image patches, reaching 83.2% ImageNet-1K accuracy."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2106.08254/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":23,"sample":[{"doi":"","year":2020,"title":"UniLMv2: Pseudo- masked language models for unified language model pre-training","work_id":"75b85408-f57b-491a-83ba-55eecee65a8c","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2003,"title":"Improved Baselines with Momentum Contrastive Learning","work_id":"f275e715-bcdc-487c-bc31-6f98d8a01f5c","ref_index":2,"cited_arxiv_id":"2003.04297","is_internal_anchor":true},{"doi":"","year":2011,"title":"Exploring simple siamese representation learning","work_id":"c76a59f1-047c-4207-a011-a8a377770323","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2002,"title":"A Simple Framework for Contrastive Learning of Visual Representations","work_id":"77d995ce-c44e-4692-9c54-cf8ce771464a","ref_index":4,"cited_arxiv_id":"2002.05709","is_internal_anchor":true},{"doi":"","year":null,"title":"Emerging Properties in Self-Supervised Vision Transformers","work_id":"6b124bd1-c9f1-4251-96c1-2683f7f17a64","ref_index":5,"cited_arxiv_id":"2104.14294","is_internal_anchor":true}],"resolved_work":23,"snapshot_sha256":"731d8af44338458a7b1e2dce8c54bf922a3d05f59aa5d678ff819f300963bd62","internal_anchors":8},"formal_canon":{"evidence_count":2,"snapshot_sha256":"18e806fdd87e9c859b4cded61857077d777334504d5e7e737035fff7b76a7590"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2106.08254","created_at":"2026-07-05T04:54:14.716320+00:00"},{"alias_kind":"arxiv_version","alias_value":"2106.08254v2","created_at":"2026-07-05T04:54:14.716320+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2106.08254","created_at":"2026-07-05T04:54:14.716320+00:00"},{"alias_kind":"pith_short_12","alias_value":"WYOEE2ZBWHIB","created_at":"2026-07-05T04:54:14.716320+00:00"},{"alias_kind":"pith_short_16","alias_value":"WYOEE2ZBWHIBD7DA","created_at":"2026-07-05T04:54:14.716320+00:00"},{"alias_kind":"pith_short_8","alias_value":"WYOEE2ZB","created_at":"2026-07-05T04:54:14.716320+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":74,"internal_anchor_count":74,"sample":[{"citing_arxiv_id":"2606.19161","citing_title":"HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2606.17256","citing_title":"Contrastive Action-Image Pre-training for Visuomotor Control","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01918","citing_title":"Zeus: Towards Tuning-Free Foundation Model for Time Series Analysis","ref_index":72,"is_internal_anchor":true},{"citing_arxiv_id":"2606.17030","citing_title":"Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation","ref_index":80,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01630","citing_title":"DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2607.02447","citing_title":"Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2606.13289","citing_title":"HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2606.13288","citing_title":"Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2606.11304","citing_title":"SPADE: Split-and-Delay Embeddings for Autoregressive High-Granularity Calorimeter Simulation","ref_index":47,"is_internal_anchor":true},{"citing_arxiv_id":"2606.11074","citing_title":"Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2606.11363","citing_title":"NSVQ: Mitigating Codebook Collapse by Stabilizing Encoder Drift in Vector Quantization","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2606.10967","citing_title":"Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2606.10372","citing_title":"ClinReadNet: A clinical reading-inspired network for low-dose abdominal CT image quality assessment","ref_index":47,"is_internal_anchor":true},{"citing_arxiv_id":"2606.05855","citing_title":"EEGDancer: Dynamic Emotion Latent Space Masked Modeling with Reinforcement Learning for EEG Continuous Emotion Prediction","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01145","citing_title":"A Lightweight Self-Supervised Learning Framework for Multivariate Time Series using Hierarchical-JEPA on ECG Data","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2607.00850","citing_title":"Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2606.03508","citing_title":"Structure-Guided Mixed Masked Pretraining and Spatial Continuity Regularization for Printed Circuit Board Defect Detection","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2605.00242","citing_title":"MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.27893","citing_title":"SIGMA: Bridging Structural and Distributional Gaps for Vision Foundation Model Adaptation","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.28940","citing_title":"Neural Scaling Laws for Jet Generation","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.31068","citing_title":"HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2605.31108","citing_title":"Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2604.21502","citing_title":"VFM$^{4}$SDG: Unveiling the Power of VFMs for Single-Domain Generalized Object Detection","ref_index":41,"is_internal_anchor":true},{"citing_arxiv_id":"2408.13471","citing_title":"Disentangled Generative Graph Representation Learning","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2410.21169","citing_title":"Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction","ref_index":16,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU","json":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU.json","graph_json":"https://pith.science/api/pith-number/WYOEE2ZBWHIBD7DAZTWK74NMIU/graph.json","events_json":"https://pith.science/api/pith-number/WYOEE2ZBWHIBD7DAZTWK74NMIU/events.json","paper":"https://pith.science/paper/WYOEE2ZB"},"agent_actions":{"view_html":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU","download_json":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU.json","view_paper":"https://pith.science/paper/WYOEE2ZB","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2106.08254&json=true","fetch_graph":"https://pith.science/api/pith-number/WYOEE2ZBWHIBD7DAZTWK74NMIU/graph.json","fetch_events":"https://pith.science/api/pith-number/WYOEE2ZBWHIBD7DAZTWK74NMIU/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU/action/timestamp_anchor","attest_storage":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU/action/storage_attestation","attest_author":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU/action/author_attestation","sign_citation":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU/action/citation_signature","submit_replication":"https://pith.science/pith/WYOEE2ZBWHIBD7DAZTWK74NMIU/action/replication_record"}},"created_at":"2026-07-05T04:54:14.716320+00:00","updated_at":"2026-07-05T04:54:14.716320+00:00"}