{"id":"b9bec247-b883-447d-a898-52844bbae9d3","arxiv_id":"2506.21602","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BiMark embeds multi-bit messages into LLM-generated text via bit-flip unbiased multilayer reweighting, preserving quality and enabling model-agnostic, message-agnostic detection.","lead":"BiMark is a new watermarking system that hides identification messages inside text generated by large language models while keeping the text quality close to the original. It could help regulators and platforms trace AI-generated content without degrading the user experience.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 reports per-bit accuracy as 'extraction rate'; binomial projections imply near-zero exact message recovery for 16/32-bit messages, so the claimed multi-bit capacity is not established.","rationale":"The reader's weakest assumption focuses on low-entropy text weakening the per-token bias E[G_t] = 0.5 + delta*tau_t. That is a valid boundary condition, and it is explicitly acknowledged in App. A.2. My concern is different and more direct: even in the paper's own high-entropy experimental setup, the reported per-bit extraction rates imply that exact recovery of longer messages is extremely rare. The paper defines 'extraction rate' as bit accuracy, but a multi-bit watermark's practical capacity depends on recovering the entire message; near-zero whole-message recovery for 16/32-bit payloads would undermine the central 'message embedding capacity' claim. This does not invalidate the mechanism, which clearly outperforms MPAC on the reported bit-level metric, but it makes the capacity claim conditional on the choice of metric. The pseudocode inversion and argmax typo in Algorithm 2 are presentation issues that do not change this assessment, since the reported experimental results presumably follow the corrected logic. I therefore keep the reader's CONDITIONAL verdict unchanged while sharpening the specific condition that should be verified.","tokens_in":25834,"tokens_out":19550,"duration_ms":223508,"concrete_test":"Run the released BiMark code on the Table 2 configurations (8/16/32 bits; 50/100/200/300 tokens) with at least 1,000 random messages and keys, and compute the exact-match rate, i.e., the fraction of trials in which all embedded bits are recovered correctly. If the exact-match rate is below 10% for 16-bit messages at lengths up to 100 tokens, or for 32-bit messages at any length up to 300 tokens, the 'message embedding capacity' claim should be restated as bit-level accuracy or supplemented with error correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central metric in Table 2 ('Rate') is the ratio of correctly extracted bits, not whole-message recovery. Applying the binomial to the paper's own numbers, a 32-bit message is recovered exactly with estimated probability 0.6635^32 ≈ 2e-6 at 50 tokens and 0.9022^32 ≈ 3.7% at 300 tokens; a 16-bit message is recovered exactly with probability 0.8555^16 ≈ 8% at 50 tokens. These are the paper's favorable, high-entropy C4/Llama3-8B results. A watermarking scheme that cannot recover a 16- or 32-bit payload in practice does not deliver the 'multi-bit message embedding capacity' that is a central contribution. This is distinct from the low-entropy caveat in App. A.2: the shortfall is visible in Table 2 itself. The paper should report exact-match recovery rates, or add error correction and report end-to-end message recovery, before the capacity claim can be taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes BiMark, an inference-time LLM watermarking framework that combines (i) a bit-flip unbiased reweighting rule over fixed vocabulary bipartitions, (ii) a multilayer composition of such rules, and (iii) an XOR-based one-time-pad encoding of message bits into the fair coin flips used by the reweighting. The claimed contributions are unbiased text-quality preservation, model-agnostic and message-agnostic detection, and multi-bit message capacity with extraction rates up to 30% higher than MPAC on short texts. The main theoretical result is Theorem 4.2, E_e[R_{θ,e}(P)] = P, supported by the scaling-factor constraint in Lemma 4.1 and the XOR uniform-bit argument in App. A.4. Experiments evaluate extraction rate (per-bit accuracy) and perplexity in Table 2, robustness to token substitution in Table 3, zero-bit detection in Fig. 4, downstream-task quality in Table 4, and ablations in Fig. 5.","tokens_in":26086,"tokens_out":10513,"duration_ms":111242,"significance":"The core unbiasedness mechanism is sound: the proof in App. A.1 is a direct expectation computation that does not depend on fitted parameters, and App. A.4 correctly shows that XOR with a balanced mask produces a fair coin flip regardless of the message content. The paper also provides pseudocode (App. C) and a public repository, which strengthens reproducibility. If the capacity claim can be supported by exact-message recovery metrics, BiMark would be a useful step toward unbiased multi-bit watermarking. However, as detailed below, the current evaluation metric does not establish the central multi-bit capacity claim, and the paper's own Type-II analysis in App. A.2 identifies a low-entropy limitation that is not reflected in the abstract's unqualified claims.","major_comments":[{"comment":"The 'Rate' reported in Table 2 is the ratio of correctly extracted bits, not the probability of recovering the full message. Under the independence assumption and using the paper's own favorable C4/Llama3-8B results, a 32-bit message is recovered exactly with probability about (0.6635)^32 ≈ 2×10^-6 at 50 tokens and (0.9022)^32 ≈ 3.7% at 300 tokens; a 16-bit message is recovered exactly with probability about (0.8555)^16 ≈ 8% at 50 tokens. Because 'multi-bit message embedding capacity' is one of the three central contributions claimed in the abstract and introduction, the paper either needs to report exact-match recovery rates or add error correction and report end-to-end payload recovery. As presented, the evidence in Table 2 does not establish usable multi-bit capacity.","section":"§5.1, Table 2"},{"comment":"The Type-II analysis in App. A.2 shows that the per-token bias E[G_t] is 0.5 + δτ_t in the unclipped regime and approaches 0.5 as τ_t tends to 0 or 1, so the watermark signal is weak for low-entropy or repetitive text. The extraction experiments in §5.1, however, use only C4-RealNewslike prompts with temperature-1.0 Llama3-8B generation, which are high-entropy settings. The abstract's unqualified claim that BiMark achieves message embedding capacity is therefore broader than the tested regime; the authors should either benchmark low-entropy domains (e.g., code, repetitive instructions, or greedy decoding) or explicitly scope the claim to high-entropy text.","section":"App. A.2 and §5.1"},{"comment":"The headline quantitative claims rest on single-point estimates. Tables 2 and 3 report no confidence intervals, standard deviations, or number of independent trials, so the 'up to 30% higher extraction rates' figure cannot be distinguished from noise around the point estimates. Please report means with standard deviations or confidence intervals over multiple seeds for the extraction-rate tables.","section":"§5.1, Tables 2 and 3"},{"comment":"The related-work section highlights Qu et al. (2024) as a multi-bit watermarking method with error correction, but Table 2 compares BiMark only with MPAC (and with zero-bit methods in Fig. 4). If Qu et al. is incompatible with the model-agnostic/message-agnostic setting or is not publicly available, the selection criteria in §5.1 should say so; as written, the 'state-of-the-art multi-bit watermarking methods' comparison is incomplete, and the effect of error correction on the exact-recovery shortfall in Table 2 is left unexamined.","section":"§5.1 and §2"}],"minor_comments":[{"comment":"The mapping in Algorithm 2 is inverted relative to §4.3: step 4 sets \\hat{e}_t^i = 1 when x_t ∈ V_i^0, while §4.3 defines \\hat{e}_t^i = 0 for V_i^0 and 1 for V_i^1. Please correct the algorithm.","section":"Algorithm 2, step 4"},{"comment":"Step 7 reads 'm[p] = max(M[p][0], [p][1])', which returns the vote count rather than the bit; this should be an arg max or an explicit index assignment.","section":"Algorithm 2, step 7"},{"comment":"The 'up to 30% higher extraction rates' claim should be qualified as relative per-bit accuracy for 32-bit messages at 50 tokens against MPAC(1.5), not exact-message recovery; the current phrasing invites a stronger reading.","section":"Abstract and §5.1"},{"comment":"The caption contains corrupted or missing text ('…that …LLM Original probabiltiy distribution'); please repair the rendering and the typo in 'probability'.","section":"Fig. 1 caption"},{"comment":"The voting matrices in Appendix D contain non-integer entries such as [44. 12] and [77.6, 66.4], which are inconsistent with an integer vote-count matrix; these are likely typos and should be fixed.","section":"Appendix D"},{"comment":"The paper invokes n-shot undetectability in Definition 3.1 but proves only per-token unbiasedness. Please state explicitly how the context-tracking mechanism in Algorithms 1 and 2 extends the per-token expectation to the sequence-level n-shot property.","section":"App. A.3 and Definition 3.1"}],"recommendation":"major_revision","confidential_remarks":"The most important concern is the mismatch between the evaluation metric (per-bit accuracy) and the claimed capacity (whole-message recovery). If the authors can add exact-recovery results, or add error correction and report end-to-end payload recovery, the contribution may be acceptable; without that, the central claim is overstated. The low-entropy limitation identified in App. A.2 should also be confronted experimentally before the paper can claim general message embedding capacity. The reference list is very long relative to the paper's length and contains a substantial number of self-citations; this is not disqualifying but may warrant editorial attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: BiMark's bit-flip unbiased reweighting is a genuinely new, clean mechanism, but the paper's central capacity claim does not survive contact with its own Table 2. The reported \"extraction rate\" is per-bit accuracy, not whole-message recovery. For a 32-bit payload at 50 tokens, the paper reports 66.35% bit accuracy; the implied exact-match recovery probability is about 2e-6. Even at 300 tokens, 90.22% bit accuracy gives roughly 3.7% exact recovery. For 16-bit messages, the numbers are better but still weak: 85.55% at 50 tokens implies about 8% exact recovery. The stress-test note is correct: this paper does not yet demonstrate multi-bit message embedding capacity, because no real payload gets recovered.\n\nWhat it does well: the bit-flip reweighting is elegant and correctly proved unbiased—a fair coin decides which of the two balanced partitions gets boosted, and the symmetric adjustment preserves the expected distribution. The XOR one-time-pad encoding is standard but properly applied, and the multilayer architecture is a sensible way to amplify the signal. The authors release code and run a broad enough experimental sweep (lengths, bits, paraphrase attacks, downstream tasks) to make the empirical story testable. The perplexity results in Table 4 are also reassuring: unlike Soft Red List, BiMark doesn't obviously degrade text quality.\n\nSoft spots, in proportion: the per-bit metric is not a nit; it undermines a headline contribution. The paper should report exact-match recovery or add error-correcting codes and report end-to-end message recovery. The \"up to 30%\" improvement is also underspecified—it compares BiMark against MPAC at δ=1.5, not the quality-preserving δ=1 variant, and Tables 2 and 3 lack error bars. There's a typo in Algorithm 2 (e_hat_t is assigned 1 for V_i_0, opposite the generation rule), and App. A.2's own Type-II analysis confirms the watermark weakens on low-entropy text, which further limits the practical regime.\n\nWho benefits: researchers working on LLM provenance, watermarking, and detection under regulatory pressure. The paper deserves a serious referee, not a desk reject, but it needs major revision on the capacity metric before the quantitative claims can be taken at face value.","headline":"The bit-flip reweighting is elegant and provably unbiased, but the paper's per-bit accuracy numbers do not support the claimed multi-bit message capacity.","tokens_in":26578,"tokens_out":4054,"would_cite":true,"duration_ms":40740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fair-coin-flip reweighting mechanism, stacked across multiple vocabulary partitions, lets large language models carry multi-bit provenance messages in their output text without the quality loss that biased…","keywords":["LLM watermarking","multi-bit watermarking","unbiased reweighting","model-agnostic detection","message extraction","text quality preservation","XOR one-time pad","multilayer reweighting"],"falsifier":"Run BiMark with a 32-bit message on deliberately low-entropy generations, such as forced repetition or near-deterministic continuations, and measure the extraction rate against the prediction from the paper's Type-II formula; if the rate collapses toward 50% as the average $\\tau_t$ nears 0 or 1, the coexistence of message capacity and quality holds only for high-entropy text.","tokens_in":25692,"feed_emoji":"💧","tokens_out":8621,"duration_ms":83827,"temperature":0.7,"pith_summary":"The paper claims that BiMark is a watermarking framework that combines, for the first time, unbiased text generation with multi-bit message embedding, model-agnostic detection, and message-agnostic extraction. The motivating problem is that existing schemes trade one of these properties for another: biased methods like Soft Red List and MPAC sacrifice text quality for detection strength, while unbiased methods preserve quality but either cannot carry messages or require access to model probabilities. BiMark's answer is a bit-flip unbiased reweighting function, a multilayer version of it, and an XOR-based one-time pad that turns message bits into fair coin flips. If the claim is right, practical deployment becomes easier because watermarked text is statistically indistinguishable in quality from unwatermarked text while still carrying provenance metadata, and detection works on short texts where earlier multi-bit methods struggle. The experiments report up to 30% higher extraction rates than MPAC on 50-token texts and downstream summarization and translation scores matching unwatermarked text.","feed_headline":"BiMark hides multi-bit watermarks in LLM text with quality intact","feed_subtitle":"A fair-coin reweighting trick boosts short-text message extraction by up to 30% over MPAC.","key_machinery":"The load-bearing object is the bit-flip unbiased reweighting function $R_{\\theta,e}$: a token-counting reweighting that splits the vocabulary into two equal halves $V_0,V_1$, flips a fair coin $e$, multiplies the probability of tokens in one half by $(1+\\delta_1)$ and the other by $(1-\\delta_0)$, and thereby leaves the expected distribution unchanged while making the direction of the bias detectable from token membership alone. A sequence of $d$ independent such functions composes into the multilayer reweighting mechanism, and an XOR-enhanced position allocation hides message bits by setting each layer's coin flip to $m[p] \\oplus b_i$, where $b_i$ is a pseudorandom Bernoulli(0.5) one-time-pad bit; because XOR with a random bit is itself Bernoulli(0.5), the message content never breaks unbiasedness. Detection reconstructs the partition memberships pseudorandomly, gathers votes for each message position in an $\\ell \\times 2$ voting matrix, and recovers each bit by majority vote.","core_discovery":"On the paper's own terms, the discovery is that a symmetric probability reweighting controlled by fair coin flips can carry multi-bit messages without a quality trade-off. Instead of permanently boosting a green list, the method chooses which half of the vocabulary gets boosted by a fair coin flip $e$, and that coin flip is itself a message bit $m[p]$ XORed with a pseudorandom one-time-pad bit $b_i$; repeating this across $d$ independent vocabulary bipartitions creates a multilayer statistical fingerprint. The central identity is the unbiasedness theorem $\\mathbb{E}_e[R_{\\theta,e}(P_M)] = P_M$, which holds because each token's probability is multiplied by $(1+\\delta_1)$ or $(1-\\delta_0)$ with equal probability while $\\delta_0 = \\delta_1 \\tau/(1-\\tau)$ keeps the distribution normalized. The paper argues this preserves text quality by construction, enables model-agnostic detection because detection needs only token subset memberships, and supports message-agnostic extraction because the XOR is reversible.","pith_inferences":["Editorial extension: if the per-layer unbiasedness theorem composes to full-sequence undetectability, BiMark would match the theoretical guarantees of other unbiased schemes while adding message capacity; the paper reports perplexity and downstream task scores but does not run a formal n-shot undetectability test, so that guarantee is not yet demonstrated.","Editorial extension: the Type-II analysis predicts the watermark signal is proportional to the probability mass $\\tau_t$ of a vocabulary partition, which means low-entropy, repetitive, or near-deterministic text should be a failure mode; a stress test on such text would show how wide the method's actual deployment envelope is.","Editorial extension: the voting-matrix extraction is effectively a repetition code over layers and tokens, so wrapping the message in an error-correcting code, as earlier multi-bit work did for its own scheme, is the natural next step and could push short-text extraction rates close to 100%."],"forward_implications":["For short texts, BiMark turns provenance tagging from unreliable to workable: on 50-token outputs it reports 8-bit extraction rates of 95.26% versus 49.49% for MPAC(1), and the gap grows with message length.","Watermark strength becomes a tunable knob: increasing the number of layers or the base scaling factor $\\tilde{\\delta}$ raises detectability up to a peak, after which extra layers add noise rather than signal.","Multilayer reweighting leaves multiple independent statistical traces, so the method degrades more gracefully under synonym substitution; under a 30% substitution ratio on 100-token texts, BiMark keeps extraction rates 26.24% higher than MPAC.","Because extraction is message-agnostic, the same detection procedure recovers any embedded bit-string without enumerating the message space, which is what a practical provenance service needs for model identity, timestamps, and content metadata."],"supporting_citations":[{"why":"Defines the green-list/red-list Soft Red List framework and z-test detection that BiMark's token-counting approach extends and makes unbiased.","marker":"(Kirchenbauer et al., 2023)"},{"why":"Supplies the definitions of n-shot undetectability and unbiased reweighting that BiMark builds on.","marker":"(Hu et al., 2024)"},{"why":"MPAC is the multi-bit position-allocation baseline and provides the voting-matrix extraction method BiMark adapts.","marker":"(Yoo et al., 2024)"},{"why":"Shows how error-correction codes can be layered on multi-bit watermark extraction, a comparison point for robustness.","marker":"(Qu et al., 2024)"},{"why":"DiPmark is the unbiased reweighting baseline used in downstream quality comparisons.","marker":"(Wu et al., 2024)"},{"why":"Distortion-free sampling baseline and the source of the high-entropy dependence insight used in BiMark's Type-II analysis.","marker":"(Kuditipudi et al., 2024)"},{"why":"SynthID provides the unbiased zero-bit watermarking baseline for detection and quality experiments.","marker":"(Dathathri et al., 2024)"}],"fun_headline_variants":["BiMark: unbiased watermarking boosts LLM short-text extraction by 30%","Fair-coin reweighting embeds multi-bit messages in LLM text","Multilayer watermark delivers 30% better extraction, no quality hit","BiMark: model-agnostic multi-bit watermark with unbiased reweighting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-token statistical bias created by the reweighting, $\\mathbb{E}[G_t] = 0.5 + \\tilde{\\delta}\\tau_t$, stays large enough over the text length and layer count for majority voting to recover every message bit; the paper's own Type-II analysis shows the bias shrinks toward zero as the partition probability mass $\\tau_t$ approaches 0 or 1, so the guarantee rests on the generated text being sufficiently high-entropy.","fun_headline_variants_meta":{"raw":{"variants":["BiMark: unbiased watermarking boosts LLM short-text extraction by 30%","Fair-coin reweighting embeds multi-bit messages in LLM text","Multilayer watermark delivers 30% better extraction, no quality hit","BiMark: model-agnostic multi-bit watermark with unbiased reweighting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2119,"prompt_tokens":964,"completion_tokens":1155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1072}},"tokens_in":580,"tokens_out":1155,"duration_ms":11331,"temperature":1.0,"reasoning_tokens":1072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:45:24.498946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BiMark with a 32-bit message on deliberately low-entropy generations, such as forced repetition or near-deterministic continuations, and measure the extraction rate against the prediction from the paper's Type-II formula; if the rate collapses toward 50% as the average $\\tau_t$ nears 0 or 1, the coexistence of message capacity and quality holds only for high-entropy text.","supporting_citations":[],"review_version":1}