{"id":"82fe2cc7-a614-4db0-bf45-c62db888cee3","arxiv_id":"2411.17713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A 1B Llama Guard safety model, compressed to 4-bit weights and a 20-token output vocabulary, runs on a phone at 30+ tokens per second with English F1 slightly better than the full-precision 1B model.","lead":"Meta turned its Llama guard model, a safety filter for AI conversations, into a 440 megabyte version that runs on a phone while matching or beating a model seven times larger on some safety scores. The paper explains the compression recipe and reports the speed and accuracy trade-offs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the safety-parity summary: INT4 has better F1 in only 3 of 8 non-English languages and better FPR in none, so the claimed \"5 of 8\" and \"better F1/FPR\" statements are not supported by the paper's own data.","rationale":"The paper has real engineering value: it demonstrates that pruning, quantization, unembedding pruning, and distillation can be composed to produce a 440MB model that runs on a mobile CPU through ExecuTorch, and the model is open-sourced. The mobile throughput claim is plausible and at least partially reproducible from the linked demo. The problem is the safety-parity claim, which is the headline result. The reader correctly identified the private, unreleased test set with no uncertainty quantification as a concern, but the more immediate issue is that the paper's own Table 1 does not support the summary statements. Even before considering statistical noise or evaluation-set bias, the reported numbers contradict the claims of \"better F1 and FPR ... for 5 of 8 non-English languages\": F1 is better in only three non-English languages and FPR is better in none. This is an internal inconsistency, not a disagreement with external consensus, and it undermines the central claim directly. The correct response is to require corrected reporting and a released or licensed evaluation with confidence intervals. That is the same conditional verdict the reader reached, so I do not change the verdict, but the justification should be sharpened to the table-level contradiction. A paired bootstrap re-analysis on a released test set, or even a careful recount of Table 1, would settle whether the concern lands. I mark agreement as partial because the reader's stated weakest assumption is the private-test-set reliability, whereas my load-bearing concern is the internal inconsistency of the reported metrics, with the private set as an additional layer of uncertainty.","tokens_in":7084,"tokens_out":11364,"duration_ms":121078,"concrete_test":"Obtain the internal evaluation set (or a licensed equivalent based on the MLCommons hazard taxonomy) and, using the released model, recompute per-language F1 and FPR for Llama Guard 3-1B-INT4 and Llama Guard 3-1B with identical decoding settings, adding paired bootstrap 95% confidence intervals for the differences. Then check the paper's counts: if English delta-F1 and delta-FPR are not both significant, or if fewer than 5 non-English languages have non-negative delta-F1 with non-overlapping intervals, or if any non-English language has significantly worse FPR, then the summary's \"5 of 8\" and \"better F1/FPR\" claims fail and the headline safety-parity claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety-parity claim is not supported by the paper's own Table 1. The abstract and Section 1 claim that Llama Guard 3-1B-INT4 achieves better F1 and FPR than Llama Guard 3-1B for English and 5 of 8 non-English languages, but reading the table gives a different picture. For non-English F1: INT4 is higher in Spanish (0.855 vs 0.840), Portuguese (0.844 vs 0.798), and Vietnamese (0.825 vs 0.819), equal in Italian (0.897 vs 0.897), and lower in French (0.873 vs 0.939), German (0.847 vs 0.851), Hindi (0.782 vs 0.815), and Indonesian (0.833 vs 0.875). That is 3 strict wins and 1 tie, not 5. For non-English FPR: INT4 is never lower than Llama Guard 3-1B; it is higher in six languages and equal in Italian and Spanish. For instance, French FPR rises from 0.012 to 0.072 while French F1 drops from 0.939 to 0.873. Thus the summary statement is internally inconsistent with the reported metrics, independent of any concern about the private test set. The evaluation is also on an unreleased set with point estimates only, and the English differences are tiny (F1 0.904 vs 0.899, FPR 0.084 vs 0.090), so without confidence intervals or sample sizes even the corrected comparison cannot establish the headline \"comparable or superior\" claim. The load-bearing premise, that the open-sourced model is near-parity in safety, is therefore not verifiable from this paper, and the paper's own table contradicts the more specific claims made about it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the development and evaluation of Llama Guard 3-1B-INT4, a compressed safety guard model derived from Llama 3.2 1B. The compression pipeline consists of decoder-block and MLP-neuron pruning (16 to 12 layers, hidden dimension 8192 to 6400), 4-bit weight and 8-bit activation quantization-aware training, output-vocabulary pruning from 128k to 20 tokens, and distillation from Llama Guard 3-8B. The authors report a final model size of about 440 MB, an on-device throughput of at least 30 tokens per second and time-to-first-token at most 2.5 seconds on a Moto Razr Android phone, and F1/FPR scores on an internal MLCommons-hazard-taxonomy test set in English and eight non-English languages. The central claim is that the compressed model achieves safety moderation quality comparable to or better than Llama Guard 3-1B despite being about 7x smaller.","tokens_in":1521,"tokens_out":1712,"duration_ms":87128,"significance":"The paper's engineering contribution is concrete and useful: it demonstrates that pruning, QAT, embedding quantization, and unembedding pruning can be combined to fit a functioning Llama Guard model into a mobile-friendly 440MB footprint, and it ships the model and recipes. The ExecuTorch integration and measured mobile throughput are valuable. However, the headline safety-parity claim is not currently supported by the paper's own Table 1 as written, and the evaluation rests on a private test set with point estimates only. The work is therefore significant if the claims are corrected and the evaluation is made more transparent, but not yet established at the level claimed.","major_comments":[{"comment":"The claim that Llama Guard 3-1B-INT4 achieves 'better F1 and false positive rate (FPR) than Llama Guard 3-1B for English and 5 of 8 non-English languages' is contradicted by Table 1. For non-English F1, the INT4 model is higher only in Spanish (0.855 vs 0.840), Portuguese (0.844 vs 0.798), and Vietnamese (0.825 vs 0.819), equal in Italian (0.897 vs 0.897), and lower in French, German, Hindi, and Indonesian. That is 4 of 8 languages at best on par/higher, not 5. For FPR, INT4 is never lower than Llama Guard 3-1B in any non-English language: it is higher in six languages and equal in Italian and Spanish. Section 4's statement 'on par or higher F1 ... on 5 out of 8' and the abstract's 'comparable or superior' should be revised to match the table exactly.","section":"§1 and §4, Table 1"},{"comment":"The safety comparison is based on an unreleased internal dataset with no reported number of examples, no per-language or per-hazard breakdown, no confidence intervals, and no significance tests. The differences that support the headline are small (English F1 0.904 vs 0.899; English FPR 0.084 vs 0.090), and the exact ties in Italian (0.897 and 0.111) suggest rounding or small samples. Without uncertainty quantification or at least a description of the test-set size and construction, the claim of 'comparable or superior' safety cannot be verified by readers. Please report the evaluation-set statistics, provide CIs or error bars, or clearly restrict the claim to 'on this internal set' with appropriate caveats.","section":"§2.1 and §4, Table 1"},{"comment":"The deployment result ('Moto-Razor phone', '>= 30 token/s', '<=2.5s time-to-first-token') is reported without a measurement protocol. The input length used, number of independent runs, hardware/OS version, model artifact version, memory footprint at runtime, and whether the measurement is for prompt-only or prompt+response classification are not specified. As the second half of the title's promise ('efficient safeguard') depends on this number, the protocol should be described so the measurement is reproducible.","section":"§4"}],"minor_comments":[{"comment":"There are typos: 'Enlish' in the Section 1 bullet and 'Portugese' in the Table 1 header; also the German FPR for Llama Guard 3-1B is printed as 0.06 while other rows use three decimals.","section":"§1 and Table 1"},{"comment":"Please clarify how the 20 output logits are 'expanded back' to the original 128k outputs; state whether the remaining logits are masked, set to a constant, or otherwise handled so that the developer-facing interface is unchanged.","section":"§3.3"},{"comment":"The statement that Llama Guard 3-1B-INT4 is 'considerably better' than GPT4 for 'English as well as 7/8 non-English languages' is accurate only if 'better' means better on both F1 and FPR simultaneously; for German the FPR is worse (0.135 vs 0.123). Please make this criterion explicit.","section":"§4"},{"comment":"The phrase 'group-size of 256 values per-channel' is ambiguous; specify whether the grouping is along the input or output channel dimension. In addition, Eq. (4) is described as asymmetric dynamic quantization, but the text should state the integer range and zero-point conventions consistently with the equation.","section":"§3.2, Eqs. (3)-(4)"},{"comment":"The torchtune and torchao URLs contain 'https//' instead of 'https://', and the Elhage et al. reference misspells 'Robert' as 'Roberb'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a model-release note, and the main reproducibility bottleneck is the private evaluation set. I would ask the authors to either release the evaluation set or provide detailed statistics and corrected claims before considering acceptance. The deployment measurement also needs a protocol. I do not see evidence of circularity; the comparison to Llama Guard 3-1B is external. The recommended revision is within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick heads-up on 2411.17713. The one thing worth remembering: the unembedding pruning trick is genuinely useful, but the paper's headline safety claim is contradicted by its own Table 1.\n\nWhat's actually new: they prune the unembedding to the 20 output tokens the guard actually uses (safe, unsafe, category numbers, a few delimiters), which kills a huge chunk of the 128k-vocabulary parameter count. That's a neat, task-specific move and it's open-sourced. The whole pipeline — block/neuron pruning, QAT, 4-bit weights, 8-bit activations, distillation from the 8B teacher — is a clean engineering story, and the 440MB footprint and ~30 token/s on a Moto Razor are plausible and useful for people building on-device moderation. The limitations section is honest about adversarial and multilingual fragility.\n\nThe soft spots are real and one is load-bearing. The abstract says the INT4 model gets better F1 and FPR than Llama Guard 3-1B on English and 5 of 8 non-English languages. Table 1 says otherwise. On non-English F1, INT4 wins only Spanish, Portuguese, and Vietnamese, ties Italian, and loses French, German, Hindi, and Indonesian. That's 3 wins and a tie, not 5. On non-English FPR, it never wins; it's higher in six languages and tied in two. The English F1/FPR improvement is real but tiny (0.904 vs 0.899, 0.084 vs 0.090). So the \"comparable or superior\" summary is not supported by the paper's own numbers. Section 4 quietly changes to \"on par or higher\" but even that count is 4, not 5.\n\nBeyond the internal contradiction, the evaluation is on an internal, unreleased test set with point estimates only — no confidence intervals, no sample sizes, no per-hazard breakdown. The mobile result is one phone, no methodology. So the engineering artifact is real, but the central comparative safety claim is not verifiable from the paper as written.\n\nWho's this for: people who want to ship a compact guard model on a phone. It deserves a real referee because the artifact and trick are worth publishing, but the authors need to fix the F1/FPR claims, provide more evaluation detail, and ideally release or license the test set. I'd send it to peer review with a strong note to correct the internal inconsistency.","headline":"Nice unembedding trick, but the headline safety claim is contradicted by the paper's own Table 1.","tokens_in":8144,"tokens_out":3388,"would_cite":true,"duration_ms":31383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact 440MB version of Llama Guard 3-1B reports safety-moderation scores comparable to or better than the larger model while running on an Android phone CPU at 30+ tokens per second.","keywords":["safety guard model","INT4 quantization","quantization-aware training","pruning","knowledge distillation","output unembedding pruning","on-device inference","conversation safety moderation"],"falsifier":"Run the open-sourced 440MB model on an independent public benchmark with a few thousand labeled safe/unsafe prompt and prompt-response pairs spanning the same hazard categories. Compute F1 and FPR with bootstrap confidence intervals and compare with Llama Guard 3-1B; if the INT4 model falls materially below the larger model, or if resampling flips the sign of the difference, the headline parity claim fails. Separately, measure time-to-first-token and steady-state throughput on a different Android phone; sustained throughput below 30 tokens/s falsifies the deployment claim.","tokens_in":6879,"feed_emoji":"🛡️","tokens_out":13481,"duration_ms":117527,"temperature":0.7,"pith_summary":"Llama Guard 3-1B-INT4 is a version of the Llama Guard content-moderation model cut down to 440MB, about seven times smaller in size than the already-small Llama Guard 3-1B. The authors achieve this by pruning decoder blocks and MLP width, quantizing weights to 4 bits and activations to 8 bits, trimming the output layer from a 128k-token vocabulary to the 20 tokens the guard actually emits, and distilling from an 8B-parameter teacher. On an internal test set based on a public hazard taxonomy, the compact model reports better English F1 and false-positive rates than Llama Guard 3-1B and F1 at least as high in five of eight non-English languages. The same model runs on a commodity Android phone CPU at 30 or more tokens per second with time-to-first-token at or below 2.5 seconds. The paper's central claim is that serious safety moderation no longer requires a large, server-side model: the compressed guard can sit on the device itself.","feed_headline":"A 440MB AI safety guard runs on phones and matches a model 7x its size","feed_subtitle":"A 7x-smaller guard model with near-parity scores fits on phones","key_machinery":"The mechanism that carries the size reduction is output-layer (unembedding) pruning: since Llama Guard's classification output is limited to 20 tokens—'safe', 'unsafe', the 14 hazard-category numbers, and four formatting or extra tokens—the 128k-token output matrix can be shrunk from 262.6M parameters to 40.96k parameters ($2048 \\times 20$), saving 131.3MB under 4-bit quantization. Around it, the pipeline uses four supports: cosine-similarity-based decoder-block pruning and neuron activation-magnitude pruning (with a Monte-Carlo importance estimate over a few thousand batches); quantization-aware training with symmetric INT4 group-wise weights and asymmetric INT8 per-token dynamic activations; 4-bit embedding rounding with group-size 32; and logit distillation from an 8B teacher with cross-entropy on token outputs. The unembedding trick is what turns a 0.5GB model into a 0.4GB model and is unique to the targeted output-vocabulary constraint.","core_discovery":"The paper's central claim is that a 1.1B-parameter guard model compressed to 440MB (INT4 weights, INT8 per-token activations) can match or beat its larger 1.5B-parameter bf16 counterpart on safety classification while remaining deployable on a phone. The compression chain starts from a pretrained Llama 3.2 1B model, prunes it to 12 decoder blocks and an MLP hidden dimension of 6400, applies quantization-aware training, rounds the embedding weights to 4 bits, slices the unembedding matrix from $2048 \\times 128k$ to $2048 \\times 20$, and distills the result from an 8B-parameter teacher. On an internal test set organized around a public hazard taxonomy, the compact model reports English F1 0.904 with FPR 0.084, versus 0.899 and 0.090 for Llama Guard 3-1B, and F1 at least as high in five of eight non-English languages. On a commodity Android phone CPU it achieves at least 30 tokens per second and time-to-first-token of 2.5 seconds or less. The paper takes this as evidence that pruning, quantization, output-vocabulary trimming, and distillation compound rather than interfere.","pith_inferences":["Editorial extension: the 128k-to-20 unembedding cut is not tied to safety; any classifier with a small, fixed output vocabulary can slice its final projection layer the same way.","Editorial extension: because the reported parity is a private-test-set point estimate, the strongest external test is an independent public safety benchmark with per-hazard confidence intervals.","Editorial extension: the throughput figure is tied to one phone; a broader sweep across Android devices with different CPUs and memory bandwidths would show how much of the 30 token/s claim is device-specific."],"forward_implications":["If correct, a 440MB guard model can screen every user prompt and model response on-device, removing the need to send conversational content to a server for safety checks.","A phone CPU can keep up with live chat: at 30+ tokens per second with under 2.5 seconds to first token, the guard finishes its check within the latency budget of an interactive exchange.","The output-vocabulary pruning from 128k to 20 tokens saves roughly 131MB and is transparent to developers because the 20 outputs are re-expanded to the original 128k interface.","The combination of pruning, quantization-aware training, and distillation gives a concrete recipe for shrinking other Llama-family safety models without retraining from scratch."],"supporting_citations":[{"why":"Defines the Llama Guard task and supplies the English safety finetuning data on which the compressed model is trained.","marker":"Inan et al., 2023"},{"why":"Provides the pruning-and-distillation methodology this pipeline adapts: importance calibration, pruning, then finetuning.","marker":"Muralidharan et al., 2024"},{"why":"Contributes the decoder-block importance metric (cosine similarity between block input and output) used to choose which layers to prune.","marker":"Men et al., 2024"},{"why":"Foundational quantization-aware training approach that supports the INT4 weight / INT8 activation quantization scheme.","marker":"Krishnamoorthi, 2018"},{"why":"Extends QAT to large language models, guiding the quantization-aware training step used here.","marker":"Liu et al., 2023"},{"why":"Supplies the distillation objective used to train the student model from the teacher model's logits.","marker":"Hinton, 2015"},{"why":"Source of the Llama Guard 3-8B teacher whose logits drive distillation in the paper's experiments.","marker":"Llama Team, 2024a"},{"why":"Defines the hazard taxonomy on which the internal evaluation set is based, providing the evidence for the parity claim.","marker":"MLCommons, 2024"}],"fun_headline_variants":["440MB AI guard runs on phones, matches 7x-larger model","Tiny 440MB safety model rivals 7x bigger on phone","Phone-speed AI guard: 30 tok/s, near-parity with 7x size","Compact Llama Guard: 7x smaller, phone-ready, same safety","AI guard fits on phone: 440MB, matches 7x model's scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the internal test set built on a public hazard taxonomy is representative and large enough that the reported F1/FPR point-estimate differences—about 0.005 in English F1—are real rather than noise, since no confidence intervals, per-hazard breakdowns, or test-set sizes are given; the paper's own Limitations section adds that some hazard categories need factual, up-to-date knowledge the model may not have.","fun_headline_variants_meta":{"raw":{"variants":["440MB AI guard runs on phones, matches 7x-larger model","Tiny 440MB safety model rivals 7x bigger on phone","Phone-speed AI guard: 30 tok/s, near-parity with 7x size","Compact Llama Guard: 7x smaller, phone-ready, same safety","AI guard fits on phone: 440MB, matches 7x model's scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1626,"prompt_tokens":945,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":576}},"tokens_in":561,"tokens_out":681,"duration_ms":6183,"temperature":1.0,"reasoning_tokens":576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:55:33.938417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the open-sourced 440MB model on an independent public benchmark with a few thousand labeled safe/unsafe prompt and prompt-response pairs spanning the same hazard categories. Compute F1 and FPR with bootstrap confidence intervals and compare with Llama Guard 3-1B; if the INT4 model falls materially below the larger model, or if resampling flips the sign of the difference, the headline parity claim fails. Separately, measure time-to-first-token and steady-state throughput on a different Android phone; sustained throughput below 30 tokens/s falsifies the deployment claim.","supporting_citations":[{"cited_title":"Announcing mlcommons ai safety v0.5 proof of concept","cited_arxiv_id":null,"evidence_quote":"Defines the hazard taxonomy on which the internal evaluation set is based, providing the evidence for the parity claim."}],"review_version":1}