{"id":"6a6f10a0-0654-44a8-8087-5c7f71d76bf1","arxiv_id":"2507.07700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A partial reproduction of Vec2Text confirms that embedding inversion is real, and shows that 8-bit quantization of stored embeddings is a lightweight defense that preserves retrieval while reducing reconstruction.","lead":"This reproducibility study reruns the Vec2Text embedding-inversion method and finds that its headline results mostly hold, with notable gaps where original model checkpoints are missing. It adds new tests showing that 8-bit quantization of embeddings weakens text reconstruction while preserving retrieval quality, and that password-like strings can sometimes be recovered from embeddings.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'successfully replicate ... with only minor discrepancies' is contradicted by the paper's own Tables 1-2: OpenAI-32 BLEU drops 83.4 to 43.4, Exact 60.9 to 4.8; several OOD datasets diverge >10%; causes (missing ada-ms-32, checkpoint versions, sample sizes) are unresolved.","rationale":"The reader's CONDITIONAL verdict is appropriately calibrated. The paper is transparent about its limitations and provides value: it releases code, uses released checkpoints, and its extensions (passwords, quantization, hyperparameter sensitivity) are concrete and useful. My stress test does not find a fatal flaw in the method or in the extension experiments. The load-bearing weakness is the mismatch between the abstract's success claim and the paper's own reported numbers. The 32-token OpenAI condition is not a replication at all because the checkpoint differs; the out-of-domain section documents multiple confounds (checkpoint versions, sample sizes) but does not resolve them. This is a correctness risk for the central claim, not a matter of disagreement with community consensus. A single artifact-complete rerun would settle whether the gaps are due to missing checkpoints and sample-size choices or to genuine non-reproducibility. Because the paper's own conclusion is appropriately hedged, the fix may be editorial (align the abstract with the conclusion) or experimental (obtain missing artifacts); either way, the current version should not be accepted as fully establishing the headline replication claim. Hence I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":22590,"tokens_out":8438,"duration_ms":88916,"concrete_test":"Request from the original authors the missing ada-ms-32 checkpoint and the per-dataset sample indices/checkpoint versions used for Tables 1-2; if these are unavailable, retrain ada-ms-32 from the published MSMARCO recipe. Rerun the Table 1 OpenAI-32 condition and the Table 2 out-of-domain conditions with these exact artifacts; if any BLEU/Exact-match gap remains above 10%, the 'successfully replicate' claim should be downgraded to 'partially replicate with unresolved discrepancies'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises a successful replication with only minor discrepancies, but the paper's own comparisons do not support that. In Table 1, the OpenAI-MSMARCO condition truncated to 32 tokens shows BLEU 83.4 vs 43.4 and Exact-match 60.9 vs 4.8. Section 4.2.2 explains that the ada-ms-32 checkpoint was not released, so ada-ms-128 was used instead; that makes the comparison one between different models, not a replication of the reported condition. In Table 2, several out-of-domain results differ by far more than 10% (Signal1M 80.7 vs 57.7, NQ 32.7 vs 14.7, BioASQ 22.8 vs 8.6, SciFact 16.6 vs 9.1, TREC-News 14.5 vs 7.9). Section 5.1.2 attributes these gaps to multiple versions of ada-ms-128 and to original sample sizes as small as 90 versus the standardized 200, but no experiment is shown that closes the gaps when those variables are controlled. Even the in-domain gtr-nq-32 exact-match differs by about 18 points (40.2 vs 58.2 at 20 steps), suggesting a test-set mismatch. The conclusion's hedged 'trends align' is consistent with the data; the abstract's 'successfully replicate' is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a reproducibility study of Vec2Text (Morris et al., EMNLP 2023). Using the released inversion checkpoints gtr-nq-32 and ada-ms-128 together with the original codebase, the authors compare in-domain reconstruction results on NQ and MSMARCO and out-of-domain results on BEIR datasets against the numbers in the original paper. They additionally run three extensions: a hyperparameter sensitivity analysis (iterations and beam size) with a Pareto front, a password-reconstruction experiment, and an 8-bit embedding quantization defense (Absolute Maximum and Zeropoint). The abstract and conclusion claim successful replication with only minor discrepancies; the reported tables show large gaps in several conditions, most notably the OpenAI-MSMARCO 32-token setting and multiple out-of-domain datasets.","tokens_in":22876,"tokens_out":5130,"duration_ms":49509,"significance":"If the hedged conclusions are treated as the contribution, this is a useful independent evaluation: it shows that the released ada-ms-128 checkpoint does not reproduce the 32-token OpenAI results, that out-of-domain performance can differ substantially from the original report, and that length sensitivity is a real limitation. The extensions (quantization defense, password attack, Pareto-optimal hyperparameters) are practical, and the code is publicly released. However, the current wording of the headline claim overstates the evidence; the main value is in documenting partial or failed reproduction rather than confirming the original numbers. With an appropriately recalibrated abstract and a controlled test of the proposed explanations, the paper would be a solid contribution to the reproducibility literature.","major_comments":[{"comment":"The statement that the authors 'successfully replicate the original key results in both in-domain and out-of-domain settings, with only minor discrepancies' (Abstract) is contradicted by the paper's own tables. In Table 1, the OpenAI-MSMARCO 32-token condition reports BLEU 83.4 (original) vs 43.4 (ours) and Exact-match 60.9 vs 4.8; §4.2.2 acknowledges that the ada-ms-32 checkpoint was unavailable and ada-ms-128 was used instead, so this column is not a replication of the original condition. In Table 2, out-of-domain gaps exceed 10% on Signal1M (80.7 vs 57.7), NQ (32.7 vs 14.7), BioASQ (22.8 vs 8.6), SciFact (16.6 vs 9.1), and TREC-News (14.5 vs 7.9). The abstract's characterization of these as 'minor discrepancies' is not supported by the data.","section":"Abstract; §5.1.1 (Table 1); §5.1.2 (Table 2); §4.2.2"},{"comment":"The exact-match discrepancy for gtr-nq-32 at 20 iterative steps is substantial and in the opposite direction of the length-mismatch explanation: the original reports 40.2 while the reproduced result is 58.2, even though the BLEU scores are close (83.9 vs 83.6). This large gap likely indicates a different test sample or a different decoding protocol, yet §5.1.1 states that the gtr-nq-32 results are 'nearly identical performance' across all metrics. The authors should either identify the cause (e.g., test-set mismatch) or soften the claim to reflect the exact-match divergence.","section":"Table 1, gtr-nq-32 rows; §5.1.1"},{"comment":"The suggested explanations for the out-of-domain gaps (multiple versions of ada-ms-128 and sample sizes of 90 vs 200) are not tested. The paper does not run any experiment that measures the effect of sample size on the reported metrics, nor does it compare different checkpoints where multiple versions exist. Until such a controlled comparison is provided, the claim that the discrepancies are due to these artifacts remains a hypothesis, not a finding. The authors should add an ablation varying sample size on at least one dataset, or explicitly state that the cause is unresolved.","section":"§5.1.2"}],"minor_comments":[{"comment":"The figure content and caption are garbled in the submitted manuscript, with /uni... sequences appearing in place of the expected plot labels and text; please regenerate the figure so that the comparison is legible.","section":"Figure 3"},{"comment":"The column headers for gtr-nq-32 and ada-ms-128 are not clearly separated in the table layout; the current formatting makes it difficult to determine which Exact-match and Token F1 columns belong to which model.","section":"Table 4"},{"comment":"There is a typo in the sentence 'the out-of-domain performance of Vex2Text on several datasets'; 'Vex2Text' should be 'Vec2Text'.","section":"§5.1.2"},{"comment":"The speculation that the poor 32-token performance of ada-ms-128 'is due to specific adjustments made by Morris et al. during the training of ada-ms-128' should be either supported with evidence from the released code or training logs, or removed.","section":"§5.1.1"},{"comment":"The paper does not state whether BLEU is computed with the same tokenizer as the original paper; if a different tokenizer is used, that alone could produce non-trivial score differences and would be important to document.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a student-led reproducibility study with a useful set of negative and extension results. The main text overclaims, but the underlying material is appropriate for the venue if the abstract and conclusions are recalibrated to match the observed data. The garbled Figure 3 may be an artifact of the PDF conversion; please verify the source file before publication. The authors should also add a clear statement in the abstract that the 32-token OpenAI condition could not be replicated because the checkpoint is unavailable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a mostly honest reproducibility study of Vec2Text with two genuinely new pieces - the password reconstruction evaluation and a steps-versus-beam Pareto front - but the abstract overstates the replication success. The in-domain GTR-NQ numbers reproduce well; the OpenAI-32 condition does not (BLEU 43.4 vs 83.4, exact-match 4.8 vs 60.9), and several out-of-domain datasets miss by more than ten points. The paper itself explains why: ada-ms-32 was never released, sample sizes vary in the original, and checkpoint versions are uncertain. Those are real reasons, but they mean the headline claim of successful replication with only minor discrepancies is not supported by the paper's own evidence. The conclusion's weaker phrasing, 'trends align,' is the accurate summary.\n\nWhat is good: the authors ship code and results, clearly separate Original vs Ours, and flag missing artifacts instead of hiding them. The password experiment is a genuine extension: ada-ms-128 recovers 36% of easy passwords, 22% of medium, and 4% of hard, which is a concrete risk demonstration. The Pareto analysis of iterative steps versus beam width is practically useful. The quantization defense is not new (the authors acknowledge Zhuang et al.), but their variant on five BEIR datasets shows nDCG@10 roughly flat while BLEU drops sharply, so it serves as a decent independent confirmation.\n\nSoft spots, in proportion. First and biggest: the abstract/claim mismatch. The internal tables contradict 'successfully replicate' as written; this is fixable by rephrasing to 'reproduce in part' and making the failed conditions explicit up front. Second, no variance reporting anywhere. Password and quantization tables have no error bars or multiple seeds, and given the small sample sizes and observed instability, the 36/22/4 numbers could be less robust than they look. Third, the quantization defense assumes a non-adaptive adversary; the paper acknowledges this, but the conclusion should carry the same caveat. Fourth, the cause of the 32-token failure is hypothesized, not tested. A simple experiment with a 32-token-appropriate checkpoint or a fine-tuned variant could close the gap, but it is not done. These are addressable issues, not fatal ones.\n\nWho this is for: people working on embedding privacy, dense retrieval, and reproducibility studies. It is a useful data point for the subfield, though the overclaiming means it should be read with the tables in front of you. It deserves a serious referee: the extension experiments are worth reviewing, and the replication claims need to be forced into alignment with the evidence.\n\nRecommendation: engage with it, and if you are asked to review, accept - the main work is calibrating the claims and asking for variance reporting.","headline":"Useful reproduction of Vec2Text with real extensions, but the abstract's 'successfully replicate' is stronger than the paper's own tables support.","tokens_in":23444,"tokens_out":1901,"would_cite":true,"duration_ms":22612,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This reproducibility study confirms that text embeddings can be inverted back to the original text—including passwords—and that 8-bit quantization is a simple defense that keeps retrieval quality.","keywords":["text embedding inversion","Vec2Text","privacy leakage","reproducibility study","embedding quantization","password reconstruction","Gaussian noise defense","dense retrieval"],"falsifier":"Publish the missing 32-token model checkpoint and the exact evaluation splits, then rerun the 32-token condition of the API-based encoder: if the reproduced BLEU stays near 43 rather than the original roughly 83, the exact-replication claim for that condition is disproved; reaching about 83 would confirm it.","tokens_in":22353,"feed_emoji":"🔑","tokens_out":9874,"duration_ms":96807,"temperature":0.7,"pith_summary":"This paper tries to establish that the Vec2Text embedding-inversion attack is real, reproducible, and more dangerous than the original study fully showed, while also showing that a cheap defense exists. Running the released inversion models on the original datasets, the authors match the original in-domain scores on the GTR/NQ and 128-token OpenAI/MSMARCO conditions and confirm the main out-of-domain trend. They then show the attack reaches new targets: the ada-ms-128 model recovers 36% of Easy, 22% of Medium, and 4% of Hard passwords from a password-leak corpus, texts that have no semantic structure for the generator to lean on. They also show that 8-bit quantization cuts reconstruction BLEU by roughly half to two thirds while leaving retrieval nDCG@10 almost unchanged, making it a simpler defense than tuned Gaussian noise. The caveat is that exact replication is blocked by missing artifacts, notably the 32-token OpenAI model checkpoint and uncontrolled sample sizes, so the attack's effectiveness is verified only for the released models.","feed_headline":"Replication finds text embeddings leak 36% of easy passwords","feed_subtitle":"A reproducibility study backs Vec2Text inversion and shows 8-bit quantization can blunt it","key_machinery":"The load-bearing mechanism is Vec2Text's iterative correction loop. At each step an inversion encoder-decoder (initialized from T5-base) receives a concatenated input of three MLP-projected vectors—the target embedding, the embedding of the text generated so far, and the difference between them $e - \\hat{e}^{(t)}$—together with the tokens from the previous step; the decoder emits a new text whose embedding is closer to the target. The correction term is the identity doing the work: it tells the generator which direction in embedding space its current reconstruction is wrong, so repeated steps and sequence-level beam search drive the output toward the original text. The defense experiments target exactly this signal: Gaussian noise and 8-bit quantization perturb the fine-grained embedding details the correction loop depends on, while the coarse geometry used for retrieval remains intact.","core_discovery":"On the terms of the original paper, the core discovery of this study is confirmation with scope: Vec2Text works as advertised when the released checkpoints are evaluated at their trained text lengths, and its reach extends beyond fluent sentences to secret-like strings. In-domain, gtr-nq-32 matches or slightly exceeds the reported numbers (e.g., 50-step beam search BLEU 98.5 vs 97.3; exact match 94.0 vs 92.0), and ada-ms-128 with an 81-token corpus reproduces the reported scores closely; the apparent large drop in the OpenAI-32 condition (BLEU ~43 vs ~83) is explained by the unavailable ada-ms-32 model being replaced by ada-ms-128. Out-of-domain, several BEIR datasets reproduce within 10%, while others fall short, which the paper attributes to unreleased model versions and sample sizes between 90 and 200. The extensions show that the same mechanism reconstructs password-like inputs at non-trivial rates and that both absolute-max and zeropoint 8-bit quantization reduce BLEU from roughly 36–63 to 16–27 on five BEIR datasets while nDCG@10 stays within about 0.005. The paper also establishes that inversion quality degrades sharply when input length leaves the model's training range, and that for a fixed time budget, adding iterative steps buys more BLEU than widening the beam.","pith_inferences":["The paper's own evidence implies the strong 'successfully replicate' claim should be read as scoped to the released checkpoints: the missing 32-token model means the most impressive OpenAI-32 number from the original paper is not independently verified here.","An adaptive adversary who knows the quantization scheme could train a quantization-aware inversion model or denoise the quantized embeddings; the paper names this risk but does not test it, so 8-bit quantization is better treated as risk reduction than as a guaranteed defense.","The length-sensitivity result suggests a testable defense: deploy retrieval with an embedding protocol that hides or shifts the true token-length distribution, forcing an attacker to guess the training length, which the paper shows can sharply lower BLEU.","The same machinery could be pointed at recommendation-system user embeddings; the paper flags reconstructing user behavior history as future work, and the password results suggest non-linguistic embedding targets are plausible attack surfaces."],"forward_implications":["If the reproduced attack is as strong as reported, any system that stores embeddings instead of raw text should treat those embeddings as sensitive data, especially for user-generated content.","Because both inversion models degrade on out-of-length texts, the practical risk is highest for pipelines whose text length matches the attacker's assumed training length; deliberately mismatched length distributions could reduce exposure.","The Gaussian-noise defense works but only with tuned noise levels, whereas 8-bit quantization cuts reconstruction substantially with no hyperparameter search and almost no retrieval loss, making it the more deployable mitigation.","Password reconstruction at 36% Easy, 22% Medium, and 4% Hard means even semantically opaque secrets are partially recoverable from embeddings, so services handling login data should not assume that secrecy or randomness protects embedded content.","The Pareto front gives operators a concrete rule: for a fixed compute budget, increase iterative steps before increasing beam width when reconstruction quality is the goal, or invert that ordering when defending against the attack."],"supporting_citations":[{"why":"Supplies the Vec2Text framework, the released inversion checkpoints, and every original claim the paper reproduces.","marker":"[30]"},{"why":"Earlier reproducibility study of Vec2Text in dense retrieval; the authors position their setup and quantization defense against its findings.","marker":"[54]"},{"why":"Natural Questions is the in-domain dataset for the gtr-nq-32 reproduction and length-sensitivity tests.","marker":"[16]"},{"why":"MSMARCO is the in-domain training and evaluation corpus for the ada-ms-128 reproduction and out-of-domain comparisons.","marker":"[31]"},{"why":"BEIR supplies the out-of-domain datasets used for reproducibility and for both defense experiments.","marker":"[43]"},{"why":"GTR-base is the target encoder whose embeddings the gtr-nq-32 inversion model inverts.","marker":"[32]"},{"why":"text-embeddings-ada-002 is the target encoder for the ada-ms-128 models and password experiments.","marker":"[34]"},{"why":"T5-base is the initialization for the inversion encoder-decoder, fixing the architecture and parameter count.","marker":"[39]"},{"why":"Password Strength dataset is the source of easy, medium, and hard passwords for the password-reconstruction extension.","marker":"[2]"}],"fun_headline_variants":["Repro study: text embeddings leak 36% of easy passwords","8-bit quantization blunts text embedding inversion, repro finds","Vec2Text reproduction shows password leak, quantization defends","Text embeddings leak secrets—repro study backs inversion","Password leakage from embeddings cut by 8-bit quantization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the released checkpoints and evaluation protocol are the same models that produced the original reported numbers; the paper shows this is false for the 32-token condition of the API-based encoder, where the unavailable 32-token model is replaced by the 128-token one, and sample sizes and model versions also differ in the out-of-domain tests.","fun_headline_variants_meta":{"raw":{"variants":["Repro study: text embeddings leak 36% of easy passwords","8-bit quantization blunts text embedding inversion, repro finds","Vec2Text reproduction shows password leak, quantization defends","Text embeddings leak secrets—repro study backs inversion","Password leakage from embeddings cut by 8-bit quantization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001362,"raw_usage":{"total_tokens":5623,"prompt_tokens":1138,"completion_tokens":4485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":754,"completion_tokens_details":{"reasoning_tokens":4405}},"tokens_in":754,"tokens_out":4485,"duration_ms":31263,"temperature":1.0,"reasoning_tokens":4405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:34:31.101897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Publish the missing 32-token model checkpoint and the exact evaluation splits, then rerun the 32-token condition of the API-based encoder: if the reproduced BLEU stays near 43 rather than the original roughly 83, the exact-replication claim for that condition is disproved; reaching about 83 would confirm it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"text-embeddings-ada-002 is the target encoder for the ada-ms-128 models and password experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"T5-base is the initialization for the inversion encoder-decoder, fixing the architecture and parameter count."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Password Strength dataset is the source of easy, medium, and hard passwords for the password-reconstruction extension."}],"review_version":1}