Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversations

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A compact 440MB version of Llama Guard 3-1B reports safety-moderation scores comparable to or better than the larger model while running on an Android phone CPU at 30+ tokens per second.

desk verdict Nice unembedding trick, but the headline safety claim is contradicted by the paper's own Table 1. read the letter →

arxiv 2411.17713 v1 pith:WILDKNTL submitted 2024-11-18 cs.DC cs.AI

Igor Fedorov , Kate Plawiak , Lemeng Wu , Tarek Elgamal , Naveen Suda , Eric Smith , Hongyuan Zhan , Jianfeng Chi
show 12 more authors
Yuriy Hulovatyy Kimish Patel Zechun Liu Changsheng Zhao Yangyang Shi Tijmen Blankevoort Mahesh Pasupuleti Bilge Soran Zacharie Delpierre Coudert Rachad Alao Raghuraman Krishnamoorthi Vikas Chandra
This is my paper · ORCID
classification cs.DCcs.AI
keywords safetyguardmodelINT4quantizationquantization-awaretrainingpruningknowledgedistillationoutputunembeddingon-deviceinferenceconversationmoderation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Llama Guard 3-1B-INT4 is a version of the Llama Guard content-moderation model cut down to 440MB, about seven times smaller in size than the already-small Llama Guard 3-1B. The authors achieve this by pruning decoder blocks and MLP width, quantizing weights to 4 bits and activations to 8 bits, trimming the output layer from a 128k-token vocabulary to the 20 tokens the guard actually emits, and distilling from an 8B-parameter teacher. On an internal test set based on a public hazard taxonomy, the compact model reports better English F1 and false-positive rates than Llama Guard 3-1B and F1 at least as high in five of eight non-English languages. The same model runs on a commodity Android phone CPU at 30 or more tokens per second with time-to-first-token at or below 2.5 seconds. The paper's central claim is that serious safety moderation no longer requires a large, server-side model: the compressed guard can sit on the device itself.

What carries the argument

The mechanism that carries the size reduction is output-layer (unembedding) pruning: since Llama Guard's classification output is limited to 20 tokens—'safe', 'unsafe', the 14 hazard-category numbers, and four formatting or extra tokens—the 128k-token output matrix can be shrunk from 262.6M parameters to 40.96k parameters ($2048 \times 20$), saving 131.3MB under 4-bit quantization. Around it, the pipeline uses four supports: cosine-similarity-based decoder-block pruning and neuron activation-magnitude pruning (with a Monte-Carlo importance estimate over a few thousand batches); quantization-aware training with symmetric INT4 group-wise weights and asymmetric INT8 per-token dynamic activations; 4-bit embedding rounding with group-size 32; and logit distillation from an 8B teacher with cross-entropy on token outputs. The unembedding trick is what turns a 0.5GB model into a 0.4GB model and is unique to the targeted output-vocabulary constraint.

What would settle it

Run the open-sourced 440MB model on an independent public benchmark with a few thousand labeled safe/unsafe prompt and prompt-response pairs spanning the same hazard categories. Compute F1 and FPR with bootstrap confidence intervals and compare with Llama Guard 3-1B; if the INT4 model falls materially below the larger model, or if resampling flips the sign of the difference, the headline parity claim fails. Separately, measure time-to-first-token and steady-state throughput on a different Android phone; sustained throughput below 30 tokens/s falsifies the deployment claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 1.1B-parameter guard model compressed to 440MB (INT4 weights, INT8 per-token activations) can match or beat its larger 1.5B-parameter bf16 counterpart on safety classification while remaining deployable on a phone. The compression chain starts from a pretrained Llama 3.2 1B model, prunes it to 12 decoder blocks and an MLP hidden dimension of 6400, applies quantization-aware training, rounds the embedding weights to 4 bits, slices the unembedding matrix from $2048 \times 128k$ to $2048 \times 20$, and distills the result from an 8B-parameter teacher. On an internal test set organized around a public hazard taxonomy, the compact model reports English F1 0.904 with FPR 0.084, versus 0.899 and 0.090 for Llama Guard 3-1B, and F1 at least as high in five of eight non-English languages. On a commodity Android phone CPU it achieves at least 30 tokens per second and time-to-first-token of 2.5 seconds or less. The paper takes this as evidence that pruning, quantization, output-vocabulary trimming, and distillation compound rather than interfere.

Load-bearing premise

The load-bearing premise is that the internal test set built on a public hazard taxonomy is representative and large enough that the reported F1/FPR point-estimate differences—about 0.005 in English F1—are real rather than noise, since no confidence intervals, per-hazard breakdowns, or test-set sizes are given; the paper's own Limitations section adds that some hazard categories need factual, up-to-date knowledge the model may not have.

Editorial extensions

If this is right

  • If correct, a 440MB guard model can screen every user prompt and model response on-device, removing the need to send conversational content to a server for safety checks.
  • A phone CPU can keep up with live chat: at 30+ tokens per second with under 2.5 seconds to first token, the guard finishes its check within the latency budget of an interactive exchange.
  • The output-vocabulary pruning from 128k to 20 tokens saves roughly 131MB and is transparent to developers because the 20 outputs are re-expanded to the original 128k interface.
  • The combination of pruning, quantization-aware training, and distillation gives a concrete recipe for shrinking other Llama-family safety models without retraining from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 128k-to-20 unembedding cut is not tied to safety; any classifier with a small, fixed output vocabulary can slice its final projection layer the same way.
  • Editorial extension: because the reported parity is a private-test-set point estimate, the strongest external test is an independent public safety benchmark with per-hazard confidence intervals.
  • Editorial extension: the throughput figure is tied to one phone; a broader sweep across Android devices with different CPUs and memory bandwidths would show how much of the 30 token/s claim is device-specific.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports the development and evaluation of Llama Guard 3-1B-INT4, a compressed safety guard model derived from Llama 3.2 1B. The compression pipeline consists of decoder-block and MLP-neuron pruning (16 to 12 layers, hidden dimension 8192 to 6400), 4-bit weight and 8-bit activation quantization-aware training, output-vocabulary pruning from 128k to 20 tokens, and distillation from Llama Guard 3-8B. The authors report a final model size of about 440 MB, an on-device throughput of at least 30 tokens per second and time-to-first-token at most 2.5 seconds on a Moto Razr Android phone, and F1/FPR scores on an internal MLCommons-hazard-taxonomy test set in English and eight non-English languages. The central claim is that the compressed model achieves safety moderation quality comparable to or better than Llama Guard 3-1B despite being about 7x smaller.

Significance. The paper's engineering contribution is concrete and useful: it demonstrates that pruning, QAT, embedding quantization, and unembedding pruning can be combined to fit a functioning Llama Guard model into a mobile-friendly 440MB footprint, and it ships the model and recipes. The ExecuTorch integration and measured mobile throughput are valuable. However, the headline safety-parity claim is not currently supported by the paper's own Table 1 as written, and the evaluation rests on a private test set with point estimates only. The work is therefore significant if the claims are corrected and the evaluation is made more transparent, but not yet established at the level claimed.

major comments (3)
  1. [§1 and §4, Table 1] The claim that Llama Guard 3-1B-INT4 achieves 'better F1 and false positive rate (FPR) than Llama Guard 3-1B for English and 5 of 8 non-English languages' is contradicted by Table 1. For non-English F1, the INT4 model is higher only in Spanish (0.855 vs 0.840), Portuguese (0.844 vs 0.798), and Vietnamese (0.825 vs 0.819), equal in Italian (0.897 vs 0.897), and lower in French, German, Hindi, and Indonesian. That is 4 of 8 languages at best on par/higher, not 5. For FPR, INT4 is never lower than Llama Guard 3-1B in any non-English language: it is higher in six languages and equal in Italian and Spanish. Section 4's statement 'on par or higher F1 ... on 5 out of 8' and the abstract's 'comparable or superior' should be revised to match the table exactly.
  2. [§2.1 and §4, Table 1] The safety comparison is based on an unreleased internal dataset with no reported number of examples, no per-language or per-hazard breakdown, no confidence intervals, and no significance tests. The differences that support the headline are small (English F1 0.904 vs 0.899; English FPR 0.084 vs 0.090), and the exact ties in Italian (0.897 and 0.111) suggest rounding or small samples. Without uncertainty quantification or at least a description of the test-set size and construction, the claim of 'comparable or superior' safety cannot be verified by readers. Please report the evaluation-set statistics, provide CIs or error bars, or clearly restrict the claim to 'on this internal set' with appropriate caveats.
  3. [§4] The deployment result ('Moto-Razor phone', '>= 30 token/s', '<=2.5s time-to-first-token') is reported without a measurement protocol. The input length used, number of independent runs, hardware/OS version, model artifact version, memory footprint at runtime, and whether the measurement is for prompt-only or prompt+response classification are not specified. As the second half of the title's promise ('efficient safeguard') depends on this number, the protocol should be described so the measurement is reproducible.
minor comments (5)
  1. [§1 and Table 1] There are typos: 'Enlish' in the Section 1 bullet and 'Portugese' in the Table 1 header; also the German FPR for Llama Guard 3-1B is printed as 0.06 while other rows use three decimals.
  2. [§3.3] Please clarify how the 20 output logits are 'expanded back' to the original 128k outputs; state whether the remaining logits are masked, set to a constant, or otherwise handled so that the developer-facing interface is unchanged.
  3. [§4] The statement that Llama Guard 3-1B-INT4 is 'considerably better' than GPT4 for 'English as well as 7/8 non-English languages' is accurate only if 'better' means better on both F1 and FPR simultaneously; for German the FPR is worse (0.135 vs 0.123). Please make this criterion explicit.
  4. [§3.2, Eqs. (3)-(4)] The phrase 'group-size of 256 values per-channel' is ambiguous; specify whether the grouping is along the input or output channel dimension. In addition, Eq. (4) is described as asymmetric dynamic quantization, but the text should state the integer range and zero-point conventions consistently with the equation.
  5. [References] The torchtune and torchao URLs contain 'https//' instead of 'https://', and the Elhage et al. reference misspells 'Robert' as 'Roberb'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: compression results are benchmarked against external baselines, and no reported quantity reduces to a fitted input or self-citation.

full rationale

The paper's derivation chain is empirical rather than definitional. Llama Guard 3-1B-INT4 is produced by pruning, quantization, unembedding-layer pruning, and distillation from Llama Guard 3-8B, and then evaluated on an internal test set based on the MLCommons hazard taxonomy. The reported F1 and FPR numbers are measurements on that test set, not quantities reconstructed from the compression equations or from the model's own training objective, so the central 'comparable or superior' claim does not reduce to a fit. The baseline comparison to Llama Guard 3-1B is an external model rather than the paper's own output, and the teacher model is a larger, separately trained model. The paper does cite prior work by overlapping authors, including Inan et al. for Llama Guard training data, Liu et al. for QAT, and Llama Team for the base model and teacher, but those citations are provenance for standard techniques and data and are not invoked as the proof of the safety or latency claims. The internal, unreleased test set and the absence of confidence intervals are verification concerns rather than circularity; similarly, the discrepancy between the '5 of 8' summary and Table 1 is an internal-consistency or correctness issue, not a circular-reasoning issue. No equation in the paper defines the target result in terms of its own inputs, and no fitted parameter is renamed as a prediction. Accordingly, there is no specific circular step to exhibit.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims depend on several hand-chosen compression targets and quantization settings, the suitability of the base model and teacher, and an unreleased evaluation set. None of these is independently grounded in the paper, and the pruning thresholds and data details are not disclosed. The model introduces no new physical or conceptual entities, only a new artifact.

free parameters (6)
  • Pruned decoder layers = 12 of 16
    Chosen by block importance scores and target model size; no sensitivity curve or selection criterion reported.
  • Pruned MLP hidden dimension = 6400 of 8192
    Chosen from neuron importance scores and target model size; threshold not reported.
  • Weight quantization group size = 256
    Hand-chosen for INT4 per-channel groupwise quantization; affects accuracy and hardware efficiency.
  • Embedding quantization group size = 32
    Hand-chosen for embedding rounding; accuracy impact is stated as negligible but not quantified.
  • Activation quantization bit width = 8-bit per-token dynamic
    Design choice for XNNPACK kernels; affects throughput and accuracy.
  • Output vocabulary after unembedding pruning = 20 tokens
    Chosen as the set of tokens needed for safety labels; limits the model to 14 categories and a fixed output format.
assumptions (4)
  • domain assumption Block cosine-similarity and neuron mean-squared-activation importance metrics identify components that can be removed without harming safety classification.
    Used to justify pruning in Section 3.1; not validated against task-specific safety metrics.
  • domain assumption Llama 3.2 1B is a suitable base model for a safety guard after fine-tuning.
    Training starts from this pretrained model; the paper does not compare base-model alternatives.
  • domain assumption The internal MLCommons-based test set faithfully represents real-world safety hazards across languages.
    All F1/FPR claims are computed on this private set; no dataset statistics or inter-annotator agreement are reported.
  • standard math Monte Carlo estimates of importance scores over a few thousand batches generalize.
    Equations (1) and (2) are estimated with Monte Carlo; the paper mentions only the batch count, with no variance estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversations." pith.science (2026). https://pith.science/paper/WILDKNTL

@misc{pith2026241117713,
  author       = {Pith},
  title        = {Pith review of: Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WILDKNTL}},
  note         = {Machine review of arXiv:2411.17713}
}
read the original abstract

This paper presents Llama Guard 3-1B-INT4, a compact and efficient Llama Guard model, which has been open-sourced to the community during Meta Connect 2024. We demonstrate that Llama Guard 3-1B-INT4 can be deployed on resource-constrained devices, achieving a throughput of at least 30 tokens per second and a time-to-first-token of 2.5 seconds or less on a commodity Android mobile CPU. Notably, our experiments show that Llama Guard 3-1B-INT4 attains comparable or superior safety moderation scores to its larger counterpart, Llama Guard 3-1B, despite being approximately 7 times smaller in size (440MB).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Are Sparse Feature Interventions Actually Localized? Matched Evaluation for SAE-Based Safety Control

    cs.AI 2026-07 conditional novelty 6.5 of 10

    SAE safety ablations are regime-dependent and baseline-dependent: medium-k heads can look efficient, but surface-matched dense steering often beats them and high-k collapses coherence.

  2. MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation

    cs.CR 2025-10 conditional novelty 6.0 of 10

    Special tokens that structure LLM conversations can be injected and swapped for lookalike words to bypass both built-in safety and external content filters.

  3. WebGuard: Building a Generalizable Guardrail for Web Agents

    cs.AI 2025-07 conditional novelty 6.0 of 10

    WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...

  4. Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Risk-adaptive activation steering, guided by a prototype-similarity risk score computed on the first three response tokens, substantially reduces multimodal jailbreak success rates across four MLLMs while preserving utility.

  5. Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Seven LLMs succeed only on ciphers common in pretraining data, and few-shot examples help mostly for easy ciphers, in a new 4,509-sample cryptanalysis benchmark.

  6. Withholding the Completing Chunk: Deterministic Pair-Completion Guardrails for Streaming LLM Output

    cs.CR 2026-08 conditional novelty 4.0 of 10

    Scanning the full accumulated prefix before every stream release guarantees that the chunk completing a fixed two-part lexical danger signature is withheld.

  7. Agentic Web: Weaving the Next Web with AI Agents

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages · cited by 7 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    work page 2024

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    arXiv 2013

  5. [5]

    Privileged bases in the transformer residual stream, 2023

    Nelson Elhage, Roberb Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream, 2023. https://transformer-circuits.pub/2023/privileged-basis/index.html

    work page 2023

  6. [6]

    Executorch llama android demo app

    Executorch Team . Executorch llama android demo app. https://github.com/pytorch/executorch/tree/main/examples/demo-apps/android/LlamaDemo, 2024 a

    work page 2024

  7. [7]

    Executorch llama ios demo app

    Executorch Team . Executorch llama ios demo app. https://github.com/pytorch/executorch/tree/main/examples/demo-apps/apple_ios/LLaMA, 2024 b

    work page 2024

  8. [8]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

Show all 23 references
  1. [9]

    Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. https://arxiv.org/abs/2312.06674

    2023 arXiv

  2. [10]

    Quantizing deep convolutional networks for efficient inference: A whitepaper

    Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018

    2018 arXiv

  3. [11]

    Llm-qat: Data-free quantization aware training for large language models

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023

    2023 arXiv

  4. [12]

    The llama 3 herd of models, 2024 a

    Llama Team . The llama 3 herd of models, 2024 a . https://arxiv.org/abs/2407.21783

    2024 arXiv

  5. [13]

    Meta llama guard 2

    Llama Team . Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md, 2024 b

    2024

  6. [14]

    The llama 3 family of models

    AI @ Meta Llama Team. The llama 3 family of models. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3/1B/MODEL_CARD.md, 2024

    2024

  7. [15]

    Shortgpt: Layers in large language models are more redundant than you expect, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. https://arxiv.org/abs/2403.03853

    2024 arXiv

  8. [16]

    Announcing mlcommons ai safety v0.5 proof of concept

    MLCommons . Announcing mlcommons ai safety v0.5 proof of concept. https://mlcommons.org/2024/04/mlc-aisafety-v0-5-poc/, 2024

    2024

  9. [17]

    Compact language models via pruning and knowledge distillation, 2024

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation, 2024. https://arxiv.org/abs/2407.14679

    2024 arXiv

  10. [18]

    A white paper on neural network quantization

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021

    2021 arXiv

  11. [19]

    Executorch runtime overview

    Pytorch Team . Executorch runtime overview. https://pytorch.org/executorch/stable/runtime-overview.html, 2024 a

    2024

  12. [20]

    Executorch xnnpack delegate

    Pytorch Team . Executorch xnnpack delegate. https://pytorch.org/executorch/stable/native-delegates-executorch-xnnpack-delegate.html, 2024 b

    2024

  13. [21]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  14. [22]

    torchao: Pytorch native quantization and sparsity for training and inference, October 2024

    torchao maintainers and contributors. torchao: Pytorch native quantization and sparsity for training and inference, October 2024. https//github.com/pytorch/torchao

    2024

  15. [23]

    torchtune: Pytorch's finetuning library, April 2024

    torchtune maintainers and contributors. torchtune: Pytorch's finetuning library, April 2024. https//github.com/pytorch/torchtune

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.