pith. sign in

arxiv: 2605.20193 · v1 · pith:PQM5DNHTnew · submitted 2026-04-04 · 💻 cs.CL · cs.AI· cs.LG

Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification

Aisvarya Adeseye , Jouni Isoaho , Adeyemi Adeseye This is my paper

Pith reviewed 2026-05-21 09:30 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords quantized LLMsqualitative analysisthematic extractionmulti-pass prompt verificationhallucination reductionLLaMA-3.1interview transcriptslow-bit quantization
0
0 comments X

The pith

A multi-pass prompt verification method brings 4-bit quantized LLaMA-3.1 models closer to human-coded ground truth for extracting themes from interview transcripts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how quantization to 8-bit, 4-bit, 3-bit, and 2-bit levels affects LLaMA-3.1 performance when pulling themes from 82 interview transcripts that mix expert and non-expert language. Lower-bit versions generate more hallucinations and less stable outputs, especially on vague phrasing. The authors introduce a quantization-aware multi-pass prompt verification process that walks the model through controlled checks, discards unreliable segments, and only advances once content is verified. When applied, 4-bit models show clear gains in stability and alignment with a gold-standard reference built from NVivo human coding plus corrected high-precision outputs. If the gains hold, researchers could run reliable thematic analysis on far less expensive hardware.

Core claim

The quantization-aware multi-pass prompt verification method reduces hallucinations and improves stability of low-bit quantized LLaMA-3.1 models on qualitative thematic extraction, bringing 4-bit performance closer to the gold-standard ground truth constructed from human NVivo coding and corrected BF16 output.

What carries the argument

The quantization-aware multi-pass prompt verification method, which processes each transcript through successive verification steps that remove unreliable content before advancing.

If this is right

  • 8-bit models remain closest to the gold-standard ground truth even without the verification steps.
  • 4-bit models regain stability and accuracy once the multi-pass verification is applied.
  • 3-bit and 2-bit models suffer larger accuracy losses from compression but still improve when the prompt design and verification steps are used.
  • Models using different quantization types at the same bit width produce noticeably different results on the same thematic extraction task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The verification approach could extend to other unstructured qualitative sources such as focus-group recordings or open-ended survey answers.
  • Lowering hardware requirements might allow smaller teams without GPU clusters to incorporate LLM assistance into routine thematic coding.
  • The same stepwise checks could be tested on whether they also stabilize theme frequency counts when comparing expert versus non-expert responses.

Load-bearing premise

The gold-standard ground truth built from NVivo human coding plus manually corrected high-precision model output accurately captures the true thematic content of the 82 transcripts without systematic coder bias or semantic drift.

What would settle it

Apply the multi-pass verification method to a fresh collection of transcripts and measure agreement with a new panel of human coders who had no role in building the original gold-standard reference.

Figures

Figures reproduced from arXiv: 2605.20193 by Adeyemi Adeseye, Aisvarya Adeseye, Jouni Isoaho.

Figure 1
Figure 1. Figure 1: Multi-phase Methodology [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Multi-Pass Prompt Verification Framework (Note: The numbers shown in the arrows indicates order of functional flow) before moving to the next transcript and then repeating the process. Overall, this framework ensures higher accuracy. Additionally, it reduces inconsistent outputs, improving the stability and reliability of low-bit models [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
read the original abstract

Quantized Large Language Models (LLMs) are used more often in qualitative analysis because they run fast and need fewer computing resources. This study examines how different lower bits quantization levels (8-bit, 4-bit, 3-bit, and 2-bit) and quantization types affect the performance of LLaMA-3.1 (8B) on qualitative analysis. The study uses expert and non-expert responses from 82 interview transcripts. Low-bit models often produce higher levels of hallucinations and unstable results, especially when reading non-expert language with unclear terms. To improve performance, we propose a quantization-aware multi-pass prompt verification method. This method guides the model through controlled steps that reduce hallucinations. It removes unreliable content and passes the results to the next transcript after verification, improving accuracy. To validate performance, human coders analyzed transcripts using NVivo and BF16 LLaMA. BF16 LLaMA-3.1 produced high-precision output but had semantic drift and hallucination. These errors were corrected manually. The corrected BF16 output and NVivo human coding were combined to create a gold-standard ground truth (GSGT) for thematic extraction and frequency analysis. The results show that 8-bit models stay closest to the GSGT. The 4-bit models lose accuracy but become stable when the proposed method is applied. The 3-bit and 2-bit models drop in performance because of heavy compression, but they improve with the proposed prompt design and verification. The study also finds that models at the same bit level behave differently depending on quantization type. Overall, the method helps low-resource LLMs become more stable, accurate, and suitable for qualitative research at lower cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper examines the performance of quantized LLaMA-3.1 (8B) models (8-bit, 4-bit, 3-bit, 2-bit) on qualitative thematic extraction and frequency analysis from 82 interview transcripts. It proposes a quantization-aware multi-pass prompt verification method that guides the model through controlled steps to filter unreliable content and reduce hallucinations. Performance is measured against a gold-standard ground truth (GSGT) constructed by combining NVivo human coding with manually corrected BF16 outputs. Results indicate that 8-bit models remain closest to the GSGT, while the proposed method improves stability and accuracy for 4-bit models and provides gains for 3-bit and 2-bit models; models at the same bit level also vary by quantization type.

Significance. If the evaluation concerns can be addressed, the multi-pass verification technique offers a practical, low-overhead way to stabilize quantized LLMs for qualitative research tasks involving non-expert language, potentially enabling wider use of resource-efficient models without substantial loss in thematic fidelity. The observation that quantization type modulates performance at fixed bit widths is a useful empirical contribution for model selection in applied NLP.

major comments (2)
  1. Abstract and GSGT construction section: The gold-standard ground truth combines NVivo human coding with manually corrected BF16 outputs, yet the manuscript provides no details on the correction criteria (e.g., rules for resolving semantic drift or hallucinations), inter-coder reliability statistics, blinding procedures, or decision thresholds for discarding content. Because all accuracy, stability, and frequency metrics are computed exclusively against this GSGT, the reported gains for the multi-pass method cannot be distinguished from improved imitation of the specific manual corrections; this is load-bearing for the central claim that the method brings 4-bit performance closer to true thematic content.
  2. Results and evaluation sections: No statistical tests (e.g., paired t-tests or bootstrap confidence intervals) are reported for the claimed improvements in accuracy or stability across quantization levels, nor are exact prompt templates or verification decision rules supplied. Without these, it is impossible to assess whether the multi-pass gains are robust or reproducible, undermining the conclusion that the method makes low-bit models suitable for qualitative analysis.
minor comments (1)
  1. The abstract states that 'BF16 LLaMA-3.1 produced high-precision output but had semantic drift and hallucination' without quantifying the rate or providing examples; adding a small table of representative error types would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important areas for improving transparency and rigor in our evaluation of the multi-pass prompt verification method. We address each major comment below and will make revisions to strengthen the paper.

read point-by-point responses

  1. Referee: Abstract and GSGT construction section: The gold-standard ground truth combines NVivo human coding with manually corrected BF16 outputs, yet the manuscript provides no details on the correction criteria (e.g., rules for resolving semantic drift or hallucinations), inter-coder reliability statistics, blinding procedures, or decision thresholds for discarding content. Because all accuracy, stability, and frequency metrics are computed exclusively against this GSGT, the reported gains for the multi-pass method cannot be distinguished from improved imitation of the specific manual corrections; this is load-bearing for the central claim that the method brings 4-bit performance closer to true thematic content.

    Authors: We agree that additional details on GSGT construction are required for full transparency. In the revised manuscript, we will add a dedicated subsection describing the manual correction criteria (including specific rules for identifying and resolving semantic drift and hallucinations), inter-coder reliability statistics (e.g., Cohen's kappa computed between the human coders), blinding procedures employed during the correction process, and the decision thresholds used for retaining or discarding content. Regarding the concern that performance gains may reflect imitation of corrections rather than true thematic fidelity: the GSGT is explicitly constructed as a hybrid of independent NVivo human coding and corrected BF16 outputs to serve as the most reliable available proxy for thematic content. The multi-pass method targets quantization-induced instability and hallucinations, and its improvements are evaluated against this combined standard. We will expand the discussion to explicitly address this limitation of the GSGT as a proxy and clarify how the method's benefits extend beyond mere alignment with the corrections. revision: yes

  2. Referee: Results and evaluation sections: No statistical tests (e.g., paired t-tests or bootstrap confidence intervals) are reported for the claimed improvements in accuracy or stability across quantization levels, nor are exact prompt templates or verification decision rules supplied. Without these, it is impossible to assess whether the multi-pass gains are robust or reproducible, undermining the conclusion that the method makes low-bit models suitable for qualitative analysis.

    Authors: We concur that statistical tests and full reproducibility details are necessary. In the revised manuscript, we will incorporate statistical analyses including paired t-tests and bootstrap confidence intervals to evaluate the significance and robustness of improvements in accuracy and stability metrics across quantization levels. We will also add the exact prompt templates and verification decision rules (including step-by-step criteria for content filtering) to a new appendix, enabling full replication of the experiments. revision: yes

Circularity Check

0 steps flagged

No circularity detected in empirical evaluation chain

full rationale

The paper is an empirical study proposing a multi-pass prompt verification method for quantized LLMs and evaluating it on thematic extraction from 82 transcripts. Performance is measured by closeness to a GSGT constructed from NVivo human coding plus manual corrections to BF16 outputs. This setup does not reduce any claimed result to its inputs by construction: there are no equations, no fitted parameters renamed as predictions, no self-citation load-bearing uniqueness theorems, and no ansatz smuggled via prior work. The GSGT incorporates independent human coding alongside author corrections for semantic drift, but the evaluation remains a standard comparison against a fixed reference rather than a tautological match. The derivation chain (method design → application to low-bit models → metric computation vs. GSGT) is self-contained and externally falsifiable via the human-coded component; no step equates the output to the input definitionally.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper relies on the assumption that a hybrid human-plus-corrected-BF16 reference constitutes an unbiased ground truth and that the multi-pass verification steps do not themselves introduce new selection biases. No free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption Human coders using NVivo plus manual correction of BF16 output produce a reliable gold-standard ground truth for thematic content.
    Invoked when constructing the GSGT used to evaluate all quantized models.

pith-pipeline@v0.9.0 · 5852 in / 1395 out tokens · 39642 ms · 2026-05-21T09:30:18.562953+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 2 internal anchors

  1. [1]

    Procedia Comput

    Adeseye, A., Isoaho, J., Mohammad, T.: LLM-assisted qualitative data analysis: security and privacy concerns in gamified workforce studies. Procedia Comput. Sci. 257, 60–67 (2025)

    work page 2025

  2. [2]

    Bano, M., Hoda, R., Zowghi, D., Treude, C.: Large language models for qualitative research in software engineering: exploring opportunities and challenges. Autom. Softw. Eng. 31(1), Art. 8 (2023)

    work page 2023

  3. [3]

    Sociologica 18(2), 145–168 (2024)

    Rossi, L., Harrison, K., Shklovski, I.: The problems of LLM-generated data in social science research. Sociologica 18(2), 145–168 (2024). Contribution Title 21

    work page 2024

  4. [4]

    In: Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pp

    Fischer, T., Biemann, C.: Exploring large language models for qualitative data analysis. In: Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pp. 423–437. Association for Computational Linguistics (2024)

    work page 2024

  5. [5]

    Internet High

    Castellanos-Reyes, D., Olesova, L., Sadaf, A.: Transforming online learning re- search: leveraging GPT large language models for automated content analysis of cognitive presence. Internet High. Educ. 65, 101001 (2025)

    work page 2025

  6. [6]

    In: 2025 IEEE Nordic Circuits and Systems Conference (NorCAS), pp

    Adeseye, A., Isoaho, J., Virtanen, S., Tahir, M.: Efficient prompt design for resource-constrained deployment of local LLMs. In: 2025 IEEE Nordic Circuits and Systems Conference (NorCAS), pp. 1–7. IEEE (2025)

    work page 2025

  7. [7]

    Hallucination of Multimodal Large Language Models: A Survey

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., Shou, M.Z.: Hallucination of multimodal large language models: a survey. arXiv preprint arXiv:2404.18930 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    In: AI Revolution: Research, Ethics and Society, pp

    Adeseye, A., Isoaho, J., Tahir, M.: Performance evaluation of LLM hallucination reduction strategies for reliable qualitative analysis. In: AI Revolution: Research, Ethics and Society, pp. 142–156. Springer Nature Switzerland, Cham (2026)

    work page 2026

  9. [9]

    In: Proc

    Li, J., Wang, J., Zhang, Z., Zhao, H.: Self-prompting large language models for zero-shot open-domain QA. In: Proc. NAACL-HLT 2024 (Long Papers), pp. 296– 310 (2024)

    work page 2024

  10. [10]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: GPTQ: accurate post- training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    In: Proc

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S.: SmoothQuant: accurate and efficient post-training quantization for large language models. In: Proc. 40th Int. Conf. Machine Learning (ICML), pp. 38087–38099 (2023)

    work page 2023

  12. [12]

    In: Proc

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., Han, S.: AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In: Proc. Mach. Learn. Syst. (MLSys), vol. 6, pp. 87–100 (2024)

    work page 2024

  13. [13]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., Alistarh, D.: SpQR: a sparse- quantized representation for near-lossless LLM weight compression. arXiv preprint arXiv:2306.03078 (2023)

    work page arXiv 2023

  14. [14]

    In: Findings of ACL 2024, pp

    Jin, R., Du, J., Huang, W., Liu, W., Luan, J., Wang, B., Xiong, D.: A comprehen- sive evaluation of quantization strategies for large language models. In: Findings of ACL 2024, pp. 12186–12215 (2024)

    work page 2024

  15. [15]

    In: Efficient Natural Language and Speech Processing Workshop (NeurIPS), vol

    Li, S., Ning, X., Hong, K., Liu, T., Wang, L., Li, X., Zhong, K., Dai, G., Yang, H., Wang, Y.: LLM-MQ: mixed-precision quantization for efficient LLM deployment. In: Efficient Natural Language and Speech Processing Workshop (NeurIPS), vol. 9 (2023)

    work page 2023

  16. [16]

    ACM Trans

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. 43(2), 1–55 (2025)

    work page 2025

  17. [17]

    In: 2025 IEEE 5th Int

    Adeseye, A., Isoaho, J., Tahir, M.: Systematic prompt framework for qualitative data analysis: designing system and user prompts. In: 2025 IEEE 5th Int. Conf. Human-Machine Systems (ICHMS), pp. 229–234. IEEE (2025)

    work page 2025

  18. [18]

    In: Findings of NAACL 2025, pp

    Feng, Z., Zhang, Y., Li, H., Wu, B., Liao, J., Liu, W., Lang, J., Feng, Y., Wu, J., Liu, Z.: TEaR: improving LLM-based machine translation with systematic self- refinement. In: Findings of NAACL 2025, pp. 3922–3938 (2025). 22 A. Adeseye et al

    work page 2025

  19. [19]

    IEEE Access 13, 202143–202170 (2025)

    Xing, S., He, Y., Chen, H., Ke, W.: Incorporating LLM versus LLM into mul- timodal chain-of-thought for fine-grained evidence generation. IEEE Access 13, 202143–202170 (2025)

    work page 2025

  20. [20]

    In: Proc

    Lertvittayakumjorn, P., Kinney, D., Prabhakaran, V., Martin, D., Dev, S.: Towards geo-culturally grounded LLM generations. In: Proc. ACL 2025 (Short Papers), pp. 313–330 (2025)

    work page 2025

  21. [21]

    In: Findings of ACL 2024, pp

    Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., Yang, Y.: CriticBench: benchmarking LLMs for critique-correct reasoning. In: Findings of ACL 2024, pp. 1552–1587 (2024)

    work page 2024

  22. [22]

    arXiv preprint arXiv:2502.09061 (2025)

    Banerjee, D., Suresh, T., Ugare, S., Misailovic, S., Singh, G.: CRANE: reasoning with constrained LLM generation. arXiv preprint arXiv:2502.09061 (2025)

    work page arXiv 2025

  23. [23]

    arXiv preprint arXiv:2510.16082 (2026)

    Hossain, E., Shoeibi, M., Garibay, I., Yousefi, N.: BioGen: an evidence-grounded framework for interpreting RNA-seq gene clusters in antimicrobial resistance re- search. arXiv preprint arXiv:2510.16082 (2026)

    work page arXiv 2026

  24. [24]

    In: Proc

    Hwang, J., Park, J., Park, H., Kim, D., Park, S., Ok, J.: Retrieval-augmented generation with estimation of source reliability. In: Proc. EMNLP 2025, pp. 34279– 34303 (2025)

    work page 2025