{"id":"e3efde96-a17b-494e-a948-b51c22cb24a8","arxiv_id":"2508.18872","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper proposes LACA, a reproducible protocol in which humans build a codebook, an LLM performs deductive coding, and interrater reliability checks gate whether the LLM can code the full dataset.","lead":"This paper proposes a structured method, LACA, for using large language models to code large volumes of student feedback and other qualitative data in computing education research. It is a protocol paper with a fictitious example and open-source tools, not a validated empirical study.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LACA's reliability check is in-sample: prompt tuning on the same sample used for human-LLM IRR means full-corpus reliability is not validated.","rationale":"The reader's weakest assumption correctly identifies the sample-to-full-corpus bridge as critical, but the more specific and internal problem is that the protocol tunes the prompt on the very sample used to measure IRR, turning the reported alpha into an overfitting statistic rather than an unbiased reliability estimate. This is not merely missing empirical support; it is a design flaw in the proposed method itself, directly affecting the central 'rigorous and replicable' claim. The paper is transparent about the fictitious example and assumed IRR values, and it releases tools, which is creditable. However, a held-out validation step is absent and could be added with modest effort. The proposed test would settle whether the concern lands: if held-out IRR remains above threshold, the method's sample-based guarantee is credible; if not, the protocol must be revised. Other concerns, such as lack of original data and model choice, are secondary to this validation gap. The reader's CONDITIONAL verdict remains appropriate, so no change is recommended.","tokens_in":10978,"tokens_out":3874,"duration_ms":38681,"concrete_test":"On a real corpus (e.g., a random 5,000-abstract subset of the SIGCSE/ITiCSE data), pre-register a 20% holdout before any prompt tuning. Use the remaining 80% for human coding and prompt refinement until alpha > 0.80. Then freeze the prompt, code the holdout, have the same human coders code the holdout, and compute alpha. If holdout alpha falls below 0.80 or drops by more than 0.05 relative to the tuning-sample alpha, the LACA protocol as written does not support its full-corpus reliability claim. Repeat with temperature=0 and temperature=0.2 to bound stochastic variation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in LACA is the claim that a human-LLM IRR above 0.80 on a sample licenses running the LLM on the full dataset. As written in Section 3 (Steps 4–6), the prompt is iteratively refined on the same sample used to measure agreement: 'modify the prompt as needed depending on the IRR produced, and continue until acceptable IRR is achieved on the sample.' The final prompt is then applied to the entire dataset with no held-out validation and no monitoring for drift. This makes the reported alpha an in-sample fit statistic, not a predictive estimate. The prompt may learn idiosyncrasies of the 1,257-abstract sample (or of one stochastic run) and fail on the remaining ~11,300 abstracts. The paper's 'fatigue' discussion only covers failure to reach threshold during iteration, not degradation after threshold is reached. Because the central claim is that LACA enables rigorous, replicable analysis, the protocol needs an independent validation step to support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-assisted content analysis (LACA), a variant of content analysis in which humans construct the codebook (deductively or inductively) and an LLM performs deductive coding at scale. The authors describe a seven-step protocol covering ethical approval, codebook construction, sampling, human coding, human–LLM interrater reliability (IRR), full-corpus LLM coding, and reporting. They illustrate the protocol with a fictitious worked example based on a corpus of 12,573 computing education publication abstracts, and they provide an open repository, a Python library, and a no-code workflow tool. The central claim is that LACA enables computing education researchers to conduct larger-scale, rigorous, and replicable content analyses than would otherwise be feasible.","tokens_in":11209,"tokens_out":4262,"duration_ms":40485,"significance":"If the methodological claims are supported, LACA would be a useful contribution to CER methodology: it is grounded in established content-analysis principles, includes explicit attention to ethics and reporting, and ships reusable artifacts (the repository, library, and no-code tool) that lower the barrier to entry. The paper is also commendable for honestly discussing limitations such as 'fatigue,' stopping criteria, and uncertainty about which data types are suitable. However, the paper's central claim is not demonstrated with original empirical evidence: the worked example is explicitly fictitious and all reported IRR values are assumed. The load-bearing reliability argument also has a validation gap, as the prompt is tuned on the same sample used to measure agreement. The significance is therefore conditional on adding a proper validation step and/or providing a real worked example.","major_comments":[{"comment":"The reliability check is in-sample. Step 4 states: 'modify the prompt as needed depending on the IRR produced, and continue until acceptable IRR is achieved on the sample.' Step 6 then applies the final prompt to the entire dataset. Because the prompt is iterated on the same 1,257-abstract sample used for the reported human–LLM agreement, the resulting alpha is an in-sample fit statistic, not a predictive estimate of full-corpus reliability. No held-out sample, no monitoring for drift or distribution shift, and no handling of stochastic variation across runs are described. To support the 'rigorous and replicable' claim, the protocol should require freezing the prompt after the threshold is reached and evaluating it on an independent validation sample (ideally also reporting agreement across repeated runs and model versions).","section":"Section 3, Steps 4–6"},{"comment":"The illustrative example is entirely hypothetical, and the IRR values are assumed rather than measured. Step 3 states: 'Given the fictitious nature of our example, we assume a Krippendorff's α higher than our threshold of 0.80 was achieved'; Step 5 similarly 'assume[s] that after a series of iterations, the prompt achieves acceptable IRR.' Consequently, the abstract's claim that the paper illustrates how LACA 'could be applied in a reproducible and rigorous manner' is not supported by original data; the argument rests on cited prior work and hypotheticals. Since the central claim is that LACA is rigorous and feasible, the paper needs either a real worked example using the available repository and dataset, or a clear reframing as a proposal whose validation is left to future work, with the claims correspondingly softened.","section":"Section 3, Steps 3 and 5; Abstract"},{"comment":"The protocol does not provide methodological guidance on sampling and sample size for the reliability checks. It says to sample 'methodologically' and, in the example, to use a random sample of 1,257 abstracts (10%), but no justification is given for this size or for how representativeness is ensured. The precision of Krippendorff's α depends on the number of units and codes; a stated sample size without a rationale makes it difficult for readers to judge whether the reliability check is sufficient. The paper should provide or cite concrete guidance on determining sample size for both human-human codebook refinement and human-LLM agreement.","section":"Section 3, Step 2"}],"minor_comments":[{"comment":"There is a stray text fragment, 'interactive rebase in progress; onto b6e3daf', embedded in the middle of a sentence about future work. This appears to be an accidental editor/git artifact and should be removed.","section":"Section 5"},{"comment":"The diagram's decision points and repeat loops are hard to parse in the current PDF rendering. A numbered step list or a table accompanying the figure would make the workflow clearer, especially for readers wanting to replicate the method.","section":"Figure 1"},{"comment":"The reporting guidance lists model and version but not generation parameters (e.g., temperature, seed, sampling settings). For reproducibility, these should be reported or explicitly fixed (e.g., temperature=0).","section":"Section 3, Step 7"}],"recommendation":"major_revision","confidential_remarks":"This is a discussion paper rather than an empirical validation. The proposed method is plausible and the open-source tooling is a genuine strength, but the central claim of rigor and replicability is not yet supported because (a) the worked example is hypothetical, and (b) the reliability check is in-sample. If the authors add a held-out validation step and/or a real worked example, the paper could become a solid methodological contribution. The journal should weigh whether a proposal with a hypothetical illustration meets its bar for methods papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clear, honest methodological proposal for LLM-assisted content analysis (LACA) in computing education. What's actually new is the formalized seven-step workflow with an explicit human-LLM IRR gate, a no-code tool and open-source library, and a reporting checklist. It builds transparently on Chew et al. and grounds itself in Krippendorff and Neuendorf. The authors are upfront about the fictitious worked example and assumed IRR values, and they openly discuss limitations like \"fatigue\" and uncertainty about which data types work. That level of candor is rare and welcome.\n\nThe stress-test note is right and it's the main soft spot: Step 4 tunes the prompt on the same sample used to measure human-LLM agreement, then Step 6 applies that prompt to the full corpus with no held-out validation and no monitoring for drift. The reported alpha is an in-sample fit statistic, not a predictive estimate of full-corpus reliability. The paper even mentions \"fatigue\" during iteration but doesn't address the possibility that the prompt overfits the sample and then degrades on new data. That's a load-bearing gap for a method whose central claim is rigor and replicability.\n\nA second, related weakness is that there is no original empirical evidence. The example is explicitly fictional, so the paper's argument rests on prior literature and hypotheticals. That's acceptable for a discussion paper, but the title and abstract imply a stronger empirical warrant than the content delivers. Minor issues: the manuscript contains an unfinished artifact (a git rebase message in the main text) and a placeholder acknowledgments section; these should be cleaned before publication.\n\nThe paper is for CER researchers and qualitative methodologists who want a concrete protocol for LLM-assisted coding. It deserves a serious referee, but the referee should require an independent validation step—for example, a held-out sample or cross-validation—and ideally a real demonstration on actual data. I'd send it to peer review with major revision, and I'd cite it as a reference for a structured LACA protocol while noting the validation gap.","headline":"A transparent, well-structured protocol proposal for LLM-assisted content analysis; the main gap is that the reliability check is in-sample and the example is hypothetical, so the rigor claim isn't yet demonstrated.","tokens_in":11662,"tokens_out":1845,"would_cite":true,"duration_ms":18519,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that letting an LLM do deductive coding under a human-built codebook, with an interrater reliability gate, makes large-scale content analysis feasible and rigorous for computing education researchers.","keywords":["content analysis","large language models","deductive coding","interrater reliability","computing education research","qualitative data analysis","reproducibility","LLM-assisted content analysis"],"falsifier":"Take the same corpus and codebook, run the LLM coding on the sample ten times with the same prompt and model but different random seeds, and compute the variation in codes and in human-LLM agreement; if the across-run agreement is much lower than the sample alpha, the validation step does not guarantee reproducible full-corpus coding. Alternatively, validate on a held-out sample: compute human-LLM IRR on one sample, then check agreement on a second sample the LLM has not been tuned on; if the second sample's IRR falls below the threshold, the method's gate is insufficient.","tokens_in":10893,"feed_emoji":"🤖","tokens_out":7431,"duration_ms":62792,"temperature":0.7,"pith_summary":"LACA—LLM-assisted content analysis—is proposed as a way for computing education researchers to run content analysis on text corpora far larger than human coding teams could handle. The division of labour is the core move: humans build and iteratively refine the codebook, and a large language model applies it as a deductive coder; a human-LLM interrater reliability check on a sample decides whether the LLM proceeds to the full dataset. The authors argue this preserves the scientific-method standards of content analysis—reliability, validity, replicability—because IRR measures work the same regardless of who or what generates the codes, as long as model, prompt, and workflow are reported. If the claim holds, small teams and time-constrained educators can produce generalisable, reproducible findings from large volumes of student feedback, abstracts, or programming traces. The paper illustrates the method on 12,573 computing education abstracts using a local 27B model.","feed_headline":"LLM plus human codebook lets one researcher code 12,000 abstracts","feed_subtitle":"A seven-step method checks human–LLM agreement on a sample before the model codes the full corpus.","key_machinery":"The central object is LACA, a content-analysis variant in which humans own the codebook and an LLM performs deductive coding using that codebook as the prompt. The load-bearing mechanism is the interrater reliability gate in Step 4: two humans first reach agreement on a sample, then the LLM's codes on the same sample are compared against the human codes; only if the IRR (e.g., a modified alpha above 0.80) is reached does the analysis proceed to the full corpus. The same workflow is then rerun on all data and reported in enough detail to replicate.","core_discovery":"The paper's central claim is that LLMs can replace human coders in deductive content analysis—the stage where a fixed codebook is applied to messages—without sacrificing the methodological standards of content analysis. On the paper's view, interrater reliability measures work identically whether codes come from humans or an LLM, and detailed codebooks suit LLM capabilities, while replicability is preserved by reporting model, prompt, and dates. Consequently, LACA is proposed as a seven-step workflow: justify LLM use and obtain ethics approval, construct a codebook, have humans code a sample and compute human-human IRR, run the LLM on the same sample and compute human-LLM IRR, iterate the co","pith_inferences":["Beyond CER, the same division of labour could transfer to any field doing deductive content analysis, since LACA's steps do not depend on computing-education data.","The method's validity rests on a bridge the paper does not test: that sample-level human-LLM agreement predicts full-corpus agreement. A hold-out validation on a second human-coded sample would make that bridge observable.","Because LLMs are non-deterministic and change across versions, a practical safeguard would be to re-run a small portion of the corpus on each new model or run and recompute IRR, a step LACA does not currently mandate.","For CER specifically, LACA could enable multi-institution aggregation of student feedback and long-term trend analysis—studies previously blocked by coding capacity—though the paper only gestures at these applications."],"forward_implications":["A single researcher or small team can perform content analysis on datasets of tens of thousands of texts, removing the number of coders as the limiting factor.","IRR-based validation stays meaningful: reporting human-LLM agreement on a sample makes LLM coding auditable in the same way as human coding.","Researchers are incentivized to build more detailed, explicit codebooks, since LLM coding reliability tracks codebook clarity.","The example pipeline on 12,573 abstracts shows the method can run with a local model on a mid-range laptop, avoiding cloud privacy concerns.","The approach is deliberately limited to deductive coding; inductive code generation is left to humans."],"supporting_citations":[{"why":"Supplies the definition of content analysis and the recommended alpha threshold of 0.80 used as the agreement gate.","marker":"[14]"},{"why":"Supplies the complementary definition of content analysis and the guidance that codebooks should contain instructions, examples, and definitions.","marker":"[22]"},{"why":"Introduces the existing LLM-assisted content analysis method that this paper extends with methodological and ethical rigor.","marker":"[6]"},{"why":"Provides evidence that LLMs can perform deductive coding with accuracy comparable to or better than humans.","marker":"[19]"},{"why":"Provides evidence on where LLM qualitative coding works well and shows that LLMs struggle on categories humans also find hard to code.","marker":"[15]"},{"why":"Assesses the potential and limits of LLMs in qualitative coding, supporting the claim that detailed codebooks matter and interpretative coding is unreliable.","marker":"[16]"},{"why":"Supplies ethical guidelines for LLM use in qualitative research that LACA's reporting requirements build on.","marker":"[30]"},{"why":"Supplies the prior conference-paper corpus and theme descriptions used for the worked example's codebook.","marker":"[34]"}],"fun_headline_variants":["LLM-assisted coding lets one researcher analyze 12,000 abstracts","Seven-step method: LLM + human checks ensure rigorous coding at scale","Solo researcher + LLM: rigorous content analysis on big qualitative data","LLM replaces human coders in content analysis, but with validation","New method: LLM and human agree on sample to trust full corpus coding"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method's validity rests on the assumption that a high human-LLM agreement measured on a sample (e.g., an interrater reliability alpha above 0.80) reliably guarantees that the same LLM will code the entire dataset correctly and consistently, across runs and model versions.","fun_headline_variants_meta":{"raw":{"variants":["LLM-assisted coding lets one researcher analyze 12,000 abstracts","Seven-step method: LLM + human checks ensure rigorous coding at scale","Solo researcher + LLM: rigorous content analysis on big qualitative data","LLM replaces human coders in content analysis, but with validation","New method: LLM and human agree on sample to trust full corpus coding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1258,"prompt_tokens":727,"completion_tokens":531,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":436}},"tokens_in":471,"tokens_out":531,"duration_ms":5499,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:07:17.633767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same corpus and codebook, run the LLM coding on the sample ten times with the same prompt and model but different random seeds, and compute the variation in codes and in human-LLM agreement; if the across-run agreement is much lower than the sample alpha, the validation step does not guarantee reproducible full-corpus coding. Alternatively, validate on a held-out sample: compute human-LLM IRR on one sample, then check agreement on a second sample the LLM has not been tuned on; if the second sample's IRR falls below the threshold, the method's gate is insufficient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition of content analysis and the recommended alpha threshold of 0.80 used as the agreement gate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the complementary definition of content analysis and the guidance that codebooks should contain instructions, examples, and definitions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the existing LLM-assisted content analysis method that this paper extends with methodological and ethical rigor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that LLMs can perform deductive coding with accuracy comparable to or better than humans."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence on where LLM qualitative coding works well and shows that LLMs struggle on categories humans also find hard to code."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Assesses the potential and limits of LLMs in qualitative coding, supporting the claim that detailed codebooks matter and interpretative coding is unreliable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior conference-paper corpus and theme descriptions used for the worked example's codebook."}],"review_version":1}