{"id":"430251d2-6d1c-48fd-b951-6eba1fc47887","arxiv_id":"2412.19048","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A 2B-parameter embedding model distilled from two larger teachers achieves a 71.54 average MTEB score (No.3 as of Dec 2024), matching 7B-parameter models.","lead":"This paper describes Jasper, a 2-billion-parameter text embedding model trained to imitate two much larger teacher models through a four-stage distillation recipe. It reaches third place on the MTEB leaderboard with far fewer parameters, which matters for building cheaper and faster retrieval systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.35-point MTEB gain of Jasper over its initialization/teacher Stella is the sole evidence for the proposed distillation framework, but no ablation isolates the framework from initialization or fine-tuning effects.","rationale":"I read the paper in good faith. The reported MTEB score is an external, checkable result, and the authors have released the model, data, and training code; those are real artifacts. The load-bearing issue is causal attribution. The strongest claim contains both an empirical statement (Jasper scores 71.54 and ranks No.3) and a methodological statement (the proposed multi-stage distillation framework achieves this). The empirical statement is plausible and can be independently verified by running the released model on MTEB. The methodological statement, however, hinges on the 0.35-point difference over Stella, which is simultaneously the initialization and one of the teachers. Without ablations that neutralize this confound, the improvement cannot be credited to the framework. This matches the reader's weakest_assumption exactly. I considered whether the 'significantly outperforms other sub-2B models' claim is more load-bearing, but that is a weaker statistical claim based on two baselines and does not threaten the central contribution as directly. The lack of a contamination check is also secondary because the released model enables a post-hoc check. Thus, the appropriate verdict remains CONDITIONAL: accept the empirical result but require ablations to support the framework's contribution.","tokens_in":9190,"tokens_out":4615,"duration_ms":41838,"concrete_test":"Run a control experiment: initialize the same architecture from a non-teacher 1.5B model (e.g., Qwen2-1.5B) and apply the full 4-stage pipeline with the same teachers and data; if the resulting MTEB average is close to 71.19 (Stella) rather than 71.54, the gain is explained by Stella's initialization. Additionally, fine-tune Stella directly on the same 8M texts with the same three losses but no multi-stage schedule, and compare to Jasper; if the fine-tuned Stella matches 71.54, the multi-stage framework adds nothing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the multi-stage distillation framework (Section 2), yet the only quantitative support is the 71.54 average on MTEB versus 71.19 for Stella (Table 1). Because Jasper is initialized from stella_en_1.5B_v5 (Section 3.1) and Stella is also one of the two teachers in stages 1-2 (Section 2.3), the 0.35-point gain could plausibly result from ordinary fine-tuning on 8M texts with three losses, from the added FC layers, or from the second teacher NV-Embed-v2, rather than from the specific multi-stage/MRL design. No ablation removes Stella as teacher, changes the initialization, or compares against a single-stage distillation baseline. The paper's self-admitted limitation in Section 2.4 (no evaluation of the proposed self-distillation) reinforces that the method's components are not individually validated. Consequently, the claim that the proposed framework 'enables' the performance is not established, even though the leaderboard score itself may be correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-stage distillation framework to train Jasper, a ~2B-parameter embedding model initialized from stella_en_1.5B_v5, using NV-Embed-v2 and Stella as teachers, three loss functions (cosine, similarity, and relative similarity), MRL-based dimension reduction through additional fully connected layers, and a final vision-language alignment stage. Jasper reports an average MTEB score of 71.54 across 56 datasets, ranking No.3 on the leaderboard as of December 24, 2024, and the authors claim this is comparable to 7B-parameter models and significantly outperforms other sub-2B models. The paper also releases the model, data, and training code, and includes an instruction-robustness experiment.","tokens_in":9364,"tokens_out":4445,"duration_ms":41299,"significance":"If the performance attribution is established, this result is practically significant: a compact 2B model with near-7B-level MTEB scores, plus released model, data, and code, would be a useful contribution to the embedding-model community. The leaderboard score is an external measurement that appears accurate as reported. However, the central methodological claim is not isolated from the initialization and fine-tuning effects; the paper lacks ablations that would attribute the 0.35-point gain over Stella to the proposed framework rather than to ordinary fine-tuning or the added components.","major_comments":[{"comment":"The central claim that the proposed multi-stage distillation framework enables Jasper's performance is not supported by the experiments as presented. Jasper is initialized from stella_en_1.5B_v5 (Section 3.1), and Stella is also one of the two teachers in Stages 1–2 (Section 2.3), so the only quantitative support for the framework is the 0.35-point MTEB gap between Jasper (71.54) and Stella (71.19) in Table 1. Without ablations that (a) train from a different initialization, (b) remove Stella as a teacher, (c) compare against a single-stage distillation baseline, or (d) remove the added FC/MRL components, the gain could plausibly result from fine-tuning on the 8M-text dataset with the combined losses, from the additional NV-Embed-v2 teacher, or from the newly added fully connected layers, rather than from the specific multi-stage/MRL design. This is load-bearing because the paper's contribution is the framework (abstract, contributions, Section 2).","section":"§3.1 and Table 1"},{"comment":"The paper explicitly states that the proposed self-distillation approach for dimension reduction was not evaluated: 'we did not conduct experiments to evaluate the specific merits of this proposed approach.' This is a missing evaluation of one of the framework's components, and it prevents the reader from determining whether the MRL-based dimension reduction contributes to the reported 0.35-point gain, or whether that gain comes entirely from the stages that fine-tune the base model.","section":"§2.4"},{"comment":"The claim that Jasper 'significantly outperforms other models with fewer than 2B parameters' is based on only three sub-2B baselines in Table 1 (Stella_en_1.5B_v5, gte-Qwen2-1.5B-instruct, voyage-lite-02-instruct), one of which is the teacher/initialization of Jasper. No statistical significance test is reported, and the numerical margins (71.54 vs. 71.19, 67.16, and 67.13) are not analyzed for variance across the 56 datasets. The word 'significantly' in the abstract and conclusion is therefore not substantiated beyond point-estimate comparisons.","section":"§3.2 and Table 1"},{"comment":"Equation (3) is not fully specified, which hampers reproducibility of the core relative-similarity distillation loss. The summation condition 'ti·tj > tm·tn' is undefined in terms of which index pairs (i,j) and (m,n) range over, and the sentence after the equation says 'the total number of text pairs (i.e., N) is given by C2 C2m,' which appears garbled (likely intended as the binomial coefficient m choose 2, but the printed expression is not meaningful). This needs to be rewritten with precise index ranges and a correct closed-form expression for N.","section":"Equation (3)"}],"minor_comments":[{"comment":"In the first paragraph of the Introduction, 'METB leaderboard' should be 'MTEB leaderboard' (the benchmark is named MTEB elsewhere in the paper).","section":"Introduction"},{"comment":"Equation (2) has formatting errors: 'M SE(SX ST X , TX T T X )' should be written with proper matrix notation, such as MSE(S_X S_X^T, T_X T_X^T), and the argument order should be clarified.","section":"Equation (2)"},{"comment":"Table 3 is difficult to read because the dataset names and scores are concatenated without visible column separation; please reformat the table so that task type, dataset name, and scores are in distinct columns.","section":"Table 3"},{"comment":"Jasper is described as a '2B' model, but the stated parameter count is 1543M + 400M = 1943M, which is closer to 1.9B; please either adjust the terminology or clarify that '2B' is a rounding of the total.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"One additional consideration for the editor: stella_en_1.5B_v5, which serves as both Jasper's initialization and one of its two teachers, was released by the same author (Dun Zhang) as this manuscript. This creates a potential conflict-of-interest in the central comparison, and the editor may wish to weigh this when assessing the novelty and strength of the attribution claim. The paper's findings would be materially strengthened by an ablation study that disentangles the framework's contribution from the base model's capability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jasper is a 2B embedding model that posts a strong MTEB number (71.54), and the paper gives a fairly detailed recipe for how it was trained. The new thing is the particular 4-stage, two-teacher distillation pipeline with three losses and MRL heads, plus the released model, data, and code. That's a real artifact and a useful engineering data point.\n\nThe paper does several things well. It's transparent about the training setup: batch sizes, learning rates, checkpoints, data mix, hyperparameters are all in Section 3. It also honestly flags in Section 2.4 that the proposed self-distillation approach was not evaluated. The leaderboard score is an external measurement, not something the authors computed on a private set, so I take it as plausible.\n\nThe soft spot is real and load-bearing. Jasper is initialized from stella_en_1.5B_v5, and Stella is also one of the two teachers. The reported gain over Stella is 0.35 points (71.54 vs 71.19). That gap could come from ordinary fine-tuning on 8M texts with the three losses, from the added FC layers, or from the second teacher NV-Embed-v2. The paper doesn't isolate the contribution of the multi-stage/MRL design with any ablation. The \"significantly outperforms other models with fewer than 2B parameters\" claim rests on only two baselines, so \"significant\" is doing more work than the data supports. The stress-test note gets this right.\n\nGiven that, the central methodological claim — that the proposed framework enables the performance — is not established. The artifact is real and useful, but the scientific claim about the method needs controlled experiments. A revision that adds ablations (random or third-party init, dropping Stella as teacher, a single-stage baseline, varying losses) and more sub-2B baselines would make this a solid paper.\n\nWho is this for? Practitioners building retrieval systems who want a strong 2B model and a detailed training recipe will get value. For the research claim, it needs more work. I'd send it to review, but the reviewer should ask for those ablations and a contamination check against MTEB.\n\nRecommendation: conditional accept under heavy revision, or reject with a clear list of what's missing. The artifact deserves registration; the method claim doesn't yet.","headline":"A useful 2B embedding artifact with a detailed recipe, but the method's contribution is not isolated from its own teacher/initialization; the 0.35-point gain needs ablations to support the paper's claim.","tokens_in":9963,"tokens_out":2590,"would_cite":false,"duration_ms":22065,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jasper, a 2-billion-parameter embedding model, reaches a 71.54 average on the 56-dataset MTEB benchmark by distilling two larger teacher models through a four-stage training scheme, matching 7-billion-parameter systems.","keywords":["text embeddings","knowledge distillation","dense retrieval","Matryoshka representation learning","multi-stage distillation","MTEB","multimodal retrieval","model compression"],"falsifier":"Run the exact four-stage pipeline once from a much weaker 1.5B base model and once from Stella, and compare MTEB averages; if the weaker base reaches the same 71.5 range, the framework is the driver, while if the score collapses toward the base model's level, the gain is inherited from Stella. A second decisive check is to ablate the two teachers: train the same recipe with only the 4096-dimensional teacher and with only the 8192-dimensional teacher, and see whether the multi-teacher concatenation adds anything beyond the stronger single teacher.","tokens_in":8921,"feed_emoji":"🔎","tokens_out":9827,"duration_ms":76402,"temperature":0.7,"pith_summary":"The paper claims that a 2-billion-parameter text embedding model, Jasper, can match the retrieval quality of 7-billion-parameter models by distilling two larger teachers into it with a four-stage training scheme. On the 56-dataset MTEB benchmark, Jasper scores 71.54 on average, third on the leaderboard as of December 24, 2024, while all other models in its size class trail it. The authors trace this result to three complementary distillation losses, Matryoshka dimension reduction for the output vectors, and a final stage that aligns images to text. The practical upshot is that top-tier embedding quality can be compressed into a smaller model without supervised labels, since the teachers supply all the training signal.","feed_headline":"2B model matches 7B rivals on text embedding benchmark","feed_subtitle":"Multi-stage distillation of two larger teachers lifts Jasper to 71.54 average on 56 MTEB tasks.","key_machinery":"The engine is a weighted sum of three losses applied over four training stages. The cosine loss $L_{cosine} = \\sum_x (1 - s_x \\cdot t_x)$ pulls each student vector $s_x$ toward the concatenated normalized teacher vector $t_x$; the similarity loss $L_{sim} = \\mathrm{MSE}(S_X S_X^\\top, T_X T_X^\\top)$ aligns the student's pairwise similarity matrix with the teachers' matrices; and the relative similarity loss $L_{resim}$ uses teacher-scored pairs as soft labels, enforcing that positive pairs score above negative pairs by a margin. Stage 1 trains only the projection layer, Stage 2 also trains the last three encoder layers, Stage 3 adds three Matryoshka projection heads so vectors can be read out at low dimensions such as 512 while the high-dimensional path keeps training, and Stage 4 freezes the text side and trains the vision encoder to match caption embeddings through self-distillation. Because the teachers provide labels, all text distillation runs on unlabeled data.","core_discovery":"Jasper is initialized from the Stella 1.5B embedding model and a 400M vision encoder, then trained in four stages. In the first two stages the student's projection layer, and later the last three encoder layers, is aligned to the concatenated normalized outputs of two teacher models, one at 4096 dimensions and one at 8192 dimensions, using a three-loss objective. In the third stage, three extra projection heads are trained so the same model can emit vectors at reduced dimensions, including 512, while the original 12288-dimensional path continues training under all three losses. In the fourth stage, image embeddings are aligned to caption embeddings by self-distillation with the frozen text side as teacher. The headline result is an average MTEB score of 71.54 across 56 datasets, against 72.31 for a 7B teacher model and 71.67 for another 7B model, and the paper's claim is that the multi-stage multi-teacher distillation is what allows a 2B student to sit in that performance class.","pith_inferences":["The paper's own numbers leave open how much of the 0.35-point gain over its Stella base is systematic; repeated runs or ablations with a non-Stella base would settle whether the multi-teacher framework earns that margin.","The self-distillation dimension-reduction trick is proposed but never evaluated; if it works, it gives a label-free way to shrink any high-dimensional embedding model, not just ones trained by this pipeline.","Since Stage 4 only aligns captions and the loss oscillates, the multimodal capability is currently a proof of concept; adding hard negatives or contrastive objectives could turn it into a competitive multimodal retriever.","The same three losses could be adapted to decoder-based embedding models, where the teacher vectors would be pooled from the last-token or mean hidden states rather than from an encoder."],"forward_implications":["A 2B model can reach the same MTEB performance tier as 7B models, making frontier-grade embeddings cheaper to serve and faster at inference.","Because the three distillation losses require no supervised labels, the same recipe can scale to much larger unlabeled corpora than the 8 million texts used here.","The Matryoshka heads mean a single deployed model can serve applications needing different vector dimensions, from 512 up to 12288, without separate retraining.","The four-stage recipe transfers to other encoder-based embedding models, and the paper's self-distillation idea would let any embedding model be compressed to lower dimensions using only unlabeled text.","The caption-alignment stage provides a basic multimodal retrieval ability, and the authors identify contrastive VQA training as the natural next step."],"supporting_citations":[{"why":"This reference defines the MTEB benchmark whose 56-dataset average is the paper's headline score.","marker":"Muennighoff et al., 2023"},{"why":"This reference supplies Matryoshka Representation Learning, the method used to produce low-dimensional embeddings.","marker":"Kusupati et al., 2024"},{"why":"These references supply NV-Embed-v2, one of the two teacher models and the 7B baseline Jasper is compared against.","marker":"Lee et al., 2024; Moreira et al., 2024"},{"why":"This reference supplies FineWeb-Edu, which makes up 80% of the unsupervised text distillation data.","marker":"Lozhkov et al., 2024"},{"why":"These references establish knowledge distillation as an effective training signal for dense retrieval, motivating the paper's loss design.","marker":"Hofstätter et al., 2021; Lin et al., 2021"},{"why":"This reference supplies Infinity-MM caption pairs used in stage 4 to align the vision encoder with text embeddings.","marker":"Gu et al., 2024"},{"why":"These references supply the SigLIP vision encoder architecture that initializes the model's visual component.","marker":"Zhai et al., 2023; Alabdulmohsin et al., 2024"},{"why":"These references supply bge-en-icl, the 7B baseline that defines the performance class Jasper claims to match.","marker":"Xiao et al., 2023; Li et al., 2024"}],"fun_headline_variants":["2B Jasper rivals 7B giants on MTEB via distillation","Distilled 2B model scores 71.54, near 7B models on MTEB","Jasper 2B: multi-teacher distillation rivals 7B on MTEB","Small 2B Jasper matches 7B models on MTEB leaderboard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that Jasper's 71.54 score is produced by the proposed multi-stage distillation framework rather than inherited from the strong Stella base model, which is both Jasper's initialization and one of its two teachers.","fun_headline_variants_meta":{"raw":{"variants":["2B Jasper rivals 7B giants on MTEB via distillation","Distilled 2B model scores 71.54, near 7B models on MTEB","Jasper 2B: multi-teacher distillation rivals 7B on MTEB","Small 2B Jasper matches 7B models on MTEB leaderboard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001444,"raw_usage":{"total_tokens":5850,"prompt_tokens":1008,"completion_tokens":4842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":4754}},"tokens_in":624,"tokens_out":4842,"duration_ms":130436,"temperature":1.0,"reasoning_tokens":4754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:28.306316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact four-stage pipeline once from a much weaker 1.5B base model and once from Stella, and compare MTEB averages; if the weaker base reaches the same 71.5 range, the framework is the driver, while if the score collapses toward the base model's level, the gain is inherited from Stella. A second decisive check is to ablate the two teachers: train the same recipe with only the 4096-dimensional teacher and with only the 8192-dimensional teacher, and see whether the multi-teacher concatenation adds anything beyond the stronger single teacher.","supporting_citations":[],"review_version":1}