A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings

Anders Malthe-S{\o}renssen; Gina Passante; Halvor Tyseng; Jonas Timmann Mjaaland; Markus Fleten Kreutzer; N.G. Holmes; Rebeckah K. Fussell; Tor Ole B. Odden

REVIEW 3 major objections 6 minor 36 references

A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A few dozen hand-picked example responses, averaged into embedding centroids, can reproduce expert deductive coding of thousands of open-ended survey responses at substantial-to-almost-perfect agreement.

desk verdict The method is sensible and the evaluation is mostly careful, but the headline kappa is flattered by including the 59 category-defining responses in the test set and by auditing the human labels with the same embedding geometry. read the letter →

arxiv 2508.19836 v2 pith:UGHGA4UG submitted 2025-08-27 cs.CL physics.ed-ph

Jonas Timmann Mjaaland , Markus Fleten Kreutzer , Halvor Tyseng , Rebeckah K. Fussell , Gina Passante , N.G. Holmes , Anders Malthe-S{o}renssen , Tor Ole B. Odden This is my paper

classification cs.CLphysics.ed-ph

keywords textembeddingsfew-shotclassificationdeductivequalitativecontentanalysissurveycodingCohen'sKappascienceeducationembeddingauditfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a framework for deductive content analysis of open-ended survey responses using text embeddings. The central claim is that averaging the embedding vectors of roughly 59 hand-picked example responses into category centroids, then assigning each of 2,899 responses to the nearest centroid by cosine similarity, reproduces expert human coding at Cohen's Kappa 0.74–0.83 on an exhaustive coding task—substantial to almost-perfect agreement. The same geometry can audit previously coded datasets: flagging near-identical responses with different codes led to 153 reclassifications. Performance is weaker on selective coding with a broad 'Other' category (Kappa 0.38–0.46), but fine-tuning the embedding model with 93 labeled examples raises it to 0.50–0.53. If correct, the method offers qualitative researchers a transparent, replicable way to scale deductive coding to thousands of responses.

What carries the argument

The category centroid: the average of the embedding vectors of several representative responses chosen by a human coder for each category. Classification assigns each response to the centroid with the highest cosine similarity, a prototypical-network-like mechanism. A second mechanism is the embedding audit: pairwise cosine distance with a threshold of 0.15 flags responses that are nearly identical in content yet carry different codes, which the original coders then review and reclassify. Fine-tuning reshapes the embedding space using ordered pairs of the 93 labeled examples, with pairs labeled 1 if the two responses share a category and 0 otherwise.

What would settle it

Re-run the benchmark against the original human codes before the embedding audit, or on a fresh dataset coded by humans without any embedding assistance; if Cohen's Kappa falls below 0.74–0.83, the headline agreement depends on embedding-guided reclassification.

Watch

Extended reading notes

Core claim

The paper claims that a framework named DeSCA (called f-SCUTE in the full text) can classify open-ended survey responses by comparing each response's embedding vector to category centroids built from a handful of human-selected examples. Using 59 category-defining responses (about 2% of the dataset), six base embedding models and four variants all achieved F1 around 0.95 and Cohen's Kappa 0.74–0.83 against expert human codes for the three primary categories after removing 'Other' responses. The paper also shows that a pairwise cosine-distance audit can flag near-duplicate responses that received different human codes, leading to 153 reclassifications. On the full selective-coding task, agree

Load-bearing premise

The benchmark treats the post-audit human labels as ground truth, but those labels were partly revised by the same embedding geometry that the classifier uses, so the agreement is not measured against a fully independent human standard.

Editorial extensions

If this is right

On datasets with mutually exclusive, semantically distinct categories, a researcher can code thousands of responses from a few dozen examples at substantial-to-almost-perfect agreement.
Selective coding with an 'Other' category is harder: without fine-tuning it stays fair-to-moderate, and fine-tuning only brings it to moderate agreement.
Embedding-based auditing can surface human coding inconsistencies that keyword matching misses, and can be applied to both human-coded and machine-coded data.
Because identical responses always receive identical codes, the machine-coded result is more replicable than hand coding; data, examples, and code can be shared for independent verification.
The reported numbers come from a single physics education dataset with short, single-theme responses, so new datasets will need their own evaluation subset to gauge performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims: the Kappa values likely overstate agreement with a fully independent human standard, because the audit that corrected 153 human codes used the same embedding geometry that the classifier relies on.
For an 'Other' category, a distance threshold (leave a response uncoded if no centroid is close) may be more principled than treating 'Other' as a single centroid, since the paper's own visualization shows Other responses scattered across the embedding space.
A cost-effective workflow implied by the results: machine first-pass coding plus human review of only the responses closest to decision boundaries, rather than full manual coding of every response.
The fine-tuning gains suggest that with a few hundred labeled examples per category, embedding models could be shaped for latent or gestalt constructs, but that remains untested for psychological or motivational codes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look. The contribution is mainly an empirical benchmark: a prototypical-networks-style nearest-centroid classifier applied to 2,899 physics-education survey responses, using a handful of examples per category, and compared across ten embedding-model variants on F1, Cohen's Kappa, and MCC. That is a modest increment over the authors' own 2024 paper, but the systematic comparison and the fine-tuning study are useful. The data and code are public, the metrics are standard, and the paper openly reports the low selective-coding results (Kappa 0.38–0.46) and the single-dataset limitation. The audit of the human codes is a genuinely nice idea, and it is described transparently.

The soft spots are real but not disqualifying. First, the evaluation appears to include the 59 category-defining responses in the test set. The text says the team selected 59 responses, averaged their embeddings into centroids, and then "measure the distances between each of the 2899 embedded responses and each category centroid" without any holdout. For the exhaustive task in Table 2 they only drop the Other-coded responses, so the 37 primary-category examples are still in the test set. Those items are near-guaranteed to be correct because the centroid is built from them. The effect is small in count but concentrated in the P and S categories, which are exactly where the classifier otherwise struggles. A clean holdout evaluation, dropping those examples from the test set, is the main fix. The stress-test concern holds up.

Second, the "ground truth" is partly the method's own product: the audit reclassified 153 responses using the same embedding space the classifier uses. That does not make the comparison meaningless, but it means the kappa is not fully against an independent human standard. The authors disclose this, yet the abstract's 0.74–0.83 range is still the headline.

Minor: the fine-tuning experiment uses default hyperparameters, which the authors admit. That is fine for a proof of concept.

Net: a solid, honest paper that slightly overstates its headline result. Send it to peer review, with the clear requirement that the authors re-run the evaluation with the category-defining responses held out and, if possible, add a second independently coded dataset. A reader who wants to scale deductive coding in education research will get value from the framework and the benchmark; the actual kappa to expect on a new dataset is probably closer to the lower end of their reported range or below.

Referee Report

3 major / 6 minor

Summary. The paper introduces DeSCA, a framework for deductive semantic content analysis of open-ended survey responses using text embeddings. The method embeds responses, constructs category centroids by averaging embeddings of a small number of hand-picked example responses per category, and classifies each response by nearest centroid in cosine similarity. As a proof of concept, the authors apply the framework to 2,899 physics-education survey responses previously coded by humans into four categories (Limitations, Physics Principles, Statistics, Other). They report strong agreement with human codes on a simulated exhaustive coding task (Cohen's Kappa 0.74–0.83 across ten embedding model variants, Table 2), lower agreement on the full selective-coding task (Table 3), and improvement after fine-tuning the embedding models (Table 4). They also describe an embedding-based audit of the human-coded dataset that led to reclassification of 153 responses, and they report all main results on the resulting 'audited' labels. The paper argues that the framework is few-shot, transparent, replicable, and well-aligned with qualitative workflows.

Significance. If the headline results are taken at face value, the paper would demonstrate an important practical result: a few-shot embedding-centroid classifier can replicate expert deductive coding at substantial-to-almost-perfect agreement on a realistic survey dataset. The manuscript has several strengths: it evaluates ten embedding model variants, reports three agreement metrics (F1, Kappa, MCC), includes resampling with standard deviations, provides open data and code, and discusses the theoretical limitations of embeddings for qualitative analysis. The audit concept is also potentially valuable for quality control of hand-coded datasets. However, two methodological issues compromise the central claim as currently presented: (i) the category-defining example responses appear to be included in the test set used to compute the reported Kappa values, and (ii) the 'ground truth' labels were partly revised using the same embedding geometry that the classifier uses. Both issues directly affect the validity of the 0.74–0.83 Kappa claim, so the current evidence does not yet establish the headline conclusion in the form stated. The issues are fixable with a clean holdout evaluation and more transp

major comments (3)

[Applying the Framework; Table 2] The evaluation does not exclude the 59 category-defining responses from the test set. The text states that after selecting 59 responses to form centroids, 'we measure the distances between each of the 2899 embedded responses and each category centroid' with no exclusion, and Table 2 reports results on the full dataset minus Other-coded responses. Each category-defining response is an element of its own centroid, so its nearest-centroid classification is (near-)guaranteed correct. This is especially consequential for the small categories P (7 examples) and S (10 examples), where a handful of anchor items can materially inflate agreement. The same issue affects the fine-tuning experiment in Table 4, where the 59 category-defining responses are used as training pairs and then the full dataset is classified. The authors should re-run the evaluation with the category-defining responses held o
[Dataset Audit Using Embeddings; Tables 2–4] The 'ground truth' human labels used in the headline results were revised after an audit that flagged near-duplicate responses using cosine distance in the same embedding space as the classifier (threshold 0.15) and then reclassified 153 responses. This means the benchmark compares the method against labels that were partly constructed using the same similarity geometry. If the audit preferentially corrected cases where a human code differed from the embedding-neighborhood majority, it could inflate agreement with a nearest-centroid classifier. The pre-audit results are relegated to SI Tables S4–S8; the main text should report both pre- and post-audit metrics, or otherwise justify that the audit does not systematically favor the embedding-based method. A concrete check is to recompute Kappa on the pre-audit labels and quantify the difference.
[Abstract; 'Approximately 1–2% of the total dataset for training'] The abstract and significance statement describe the method as using 'approximately 1-2% of the total dataset for training.' If the 59 category-defining examples are also part of the evaluation set, this framing is misleading: the examples are not a held-out training set but are included in the test set. The percentage claim should be paired with a clear statement that the reported agreement is conditional on those examples being part of the scored data, and the holdout version should be the primary result. This is not merely a presentation issue; it changes the interpretation of the few-shot claim.

minor comments (6)

[General / Throughout] The framework name is inconsistent: it is called 'f-SCUTE' in the introduction, 'DeSCA' in the abstract, and 'DeFSSCUTE' in the Discussion. Please standardize to one name.
[Text Embeddings section] Typo: 'in the context edixof “river”' should read 'in the context of “river”'.
[Table 2 and Table 3] The model name 'Infloat large instruct' appears to be a typo for 'Instructor large' (the cited INSTRUCTOR model). Please correct.
[Discussion] Minor grammar issue: 'perform more poorly a more diverse dataset' should be 'perform more poorly on a more diverse dataset'.
[Evaluation methodology] The text mentions 'resampling runs' with standard deviations 'around 0.03' but does not specify what is resampled (e.g., category-defining examples, test split, or bootstrap). Please describe the resampling procedure in Methods or SI.
[Figure 3 and Figure 4] The t-SNE plots are useful for intuition, but the claim that 'regions corresponding to different codes have become significantly more distinct' after fine-tuning is based on visual inspection. Consider adding a quantitative cluster-separation measure to support this.

Circularity Check

3 steps flagged · score 6.0 of 10

Few-shot benchmark contaminated: centroid-defining responses are classified as test items, and the human ground truth was audited with the same embedding geometry.

fitted input called prediction [Section 'Applying the Framework'; results in Table 2]
"the research team who performed the original coding selected 37 responses to represent the three primary codes (L, P, and S): 20 for Limitations, 7 for Principles, and 10 for Statistics. In addition, the team selected 22 responses to define the Other category, bringing the total number of category-defining responses to 59... we measure the distances between each of the 2899 embedded responses and each category centroid."

The 59 centroid-defining responses are part of the 2,899 texts, and the exhaustive task's test set of 2,489 includes all 37 primary-code examples (7 P, 10 S). Because each centroid is the mean of those very response vectors, classifying them is equivalent to scoring the training set: their nearest centroid is their own category by construction (or near-certainly so), so they are not independent predictions. With P and S containing only 161 and 102 total responses, these training items account for 4% and 10% of the class, inflating the reported Kappa relative to a clean few-shot holdout.
other [Section 'Dataset Audit Using Embeddings']
"Therefore, we used embeddings to perform an audit of the dataset prior to testing our framework. To do so, we measured the cosine distance between all pairs of responses and flagged responses that had at least one other response within a certain distance threshold (0.15) that had received a different code... 531 of the inconsistencies were resolved by reclassifying 153 of the responses. All reported results are therefore on the audited dataset."

The benchmark 'ground truth' was not produced independently of the method: the audit flagging used pairwise cosine distance with threshold 0.15, i.e., the same similarity geometry that determines nearest-centroid classification, and 153 responses were reclassified. Thus the human labels are partly aligned to the embedding space before any agreement is computed, so the Kappa values measure agreement with a standard that the method itself helped shape. It is not a fully external criterion.

1 more flagged steps

fitted input called prediction [Section 'Performance on a Selective Coding Task Before and After Fine-tuning'; Table 4]
"we create training data by labeling the responses that were used to define the category centroids... We then fine-tuned all the six standard text embedding models and again classified the entire dataset."

The fine-tuning supervision set consists of the 59 centroid-defining responses (plus 34 from another question), and the post-fine-tuning evaluation again classifies the entire dataset, which includes those 59. The reported improvement in Kappa is therefore measured partly on the exact texts used to train the fine-tuned models, so the gain is not a clean out-of-sample result.

full rationale

The central classification rule (nearest centroid by cosine similarity) is not circular in itself: it is a standard few-shot/prototypical classifier, and the paper is transparent about the workflow. The circularity lies in the evaluation. The 59 hand-picked responses used to build the four centroids are included in the 2,899 responses that are then 'predicted,' with no holdout reported. Because the exhaustive task excludes Other, the training examples for P and S constitute roughly 4% and 10% of those classes, so the reported Kappa 0.74–0.83 in Table 2 is partly a self-classification score, not a pure few-shot generalization score. The fine-tuning experiment repeats the same leak: the 59 centroid-defining responses are used as fine-tuning training data, and the entire dataset, including those same responses, is scored afterward. A second, independent contamination is the embedding-based audit: the human labels used as ground truth were revised by flagging pairs with cosine distance below 0.15 but different codes, and 153 responses were reclassified before all results were computed. This makes the benchmark partially self-referential: the agreement is measured against labels that were adjusted using the same embedding-similarity geometry the classifier relies on. No load-bearing self-citation was found; ref. 19 from the same group is not used as the basis of the framework's validity. Correctness risk beyond circularity: generally small category sizes and a single dataset, but those are limitations, not circular steps.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing assumptions are practical: human labels as ground truth, embedding geometry as a proxy for category semantics, 59 hand-picked examples as representative of four categories, and the audit threshold of 0.15. The main free parameters are the audit threshold and fine-tuning hyperparameters.

free parameters (2)

Audit distance threshold = 0.15
Chosen to flag responses with a near-identical neighbor carrying a different code. No sensitivity analysis is reported, and the threshold determines which 669 responses were flagged and subsequently reclassified.
Fine-tuning hyperparameters = Standard library defaults
The authors state the standard parameters are 'likely suboptimal for most models'; these settings affect the selective coding improvements reported in Table 4.

assumptions (3)

domain assumption Human coding is a valid ground truth for evaluating the classifier.
The framework's performance is measured as agreement with human-assigned codes, with no independent measure of coding validity. The paper revised these codes during the audit, so the ground truth is not fully independent.
domain assumption Cosine similarity between embedding vectors reflects the semantic categories used in the coding scheme.
This is the foundational assumption of the framework (Section 'Applying the Framework'), and the paper explicitly notes that latent or gestalt constructs requiring human inference are likely poorly captured.
domain assumption Averaging a handful of example embeddings yields a valid category centroid.
Category centroids are defined as the mean of 7-22 example vectors; the paper notes result quality depends on the researcher's selection of examples.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings." pith.science (2026). https://pith.science/paper/UGHGA4UG

@misc{pith2026250819836,
  author       = {Pith},
  title        = {Pith review of: A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UGHGA4UG}},
  note         = {Machine review of arXiv:2508.19836}
}

read the original abstract

Qualitative content analysis of open-ended survey responses is a commonly used research method in science education. However, traditional coding approaches are often time-consuming and prone to inconsistency, especially when applied to large datasets. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in analysis of open-ended survey responses, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework called Deductive Semantic Content Analysis (DeSCA) that requires only a handful of examples per category to run, is transparent and replicable, and fits well with standard qualitative workflows. When benchmarked against human analysis of a physics education survey consisting of 2899 open-ended responses, the method described by our framework achieves high agreement with expert human coders across ten embeddings models on a simulated exhaustive coding task, using approximately 1-2% of the total dataset for training. The method achieves lower agreement on a complete selective coding task; this performance, however, improves with fine-tuning of the text embedding model, which can be done with a small amount of additional data. We unpack these results in terms of the theoretical assumptions of text embeddings, and further demonstrate how embeddings can be used to audit previously-analyzed datasets for coding consistency. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale.

Figures

Figures reproduced from arXiv: 2508.19836 by the authors.

**Figure 1.** Illustration of how words with similar meaning are closer to each other in the embedding space. receive a code if they do not fit well with any of the categories and/or are assigned a generic category like Other. In other cases coding is done exhaustively, where every response must be coded into one of the primary categories. This process often takes a substantial amount of time, especially with large datasets, and … view at source ↗

**Figure 2.** Illustrative figure. Representative responses for each category (bold circles) are averaged to create a category centroid (bold pentagon), which is similar to protoypical networks (18). Responses are categorized based on their proximity to the nearest centroid. classification, a given response is considered to belong to the closest category measured using the similarity metric between the embedding vector for the re… view at source ↗

**Figure 3.** 2-dimensional t-SNE plot of responses embedded with Jina small v2. Responses have been colored according to the category assigned by a human researcher. categories. Such categories are, by definition, defined by what they are not. This presents a challenge for the proposed framework, as the fundamental principle of this method is to define categories by what they are (via examples). This makes it difficult to find a… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: 2-dimensional t-SNE plot of responses embedded with Jina small v2 after fine-tuning. Responses have been colored according to the category assigned by a human researcher (after data audit). To apply this principle here, we create training data by labeling the responses…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 30 canonical work pages

[1]

(Routledge, New Y ork), 2 edition, (2013)

SR Jones, V Torres, JL Arminio, Negotiating the Complexities of Qualitative Research in Higher Education: Fundamental Elements and Issues . (Routledge, New Y ork), 2 edition, (2013)

work page 2013
[2]

X Zhai, Y Yin, JW Pellegrino, KC Haudek, LS and, Applying machine learning in science assessment: a systematic review. Stud. Sci. Educ . 56, 111–151 (2020)

work page 2020
[3]

A Ratner, CD Sa, S Wu, D Selsam, C R ´e, Data programming: Creating large training sets, quickly (2017)

work page 2017
[4]

Examination of the t-SNE plot of responses in a Table 4

The results show a substantial improvement between ten and twenty percent on both Kappa and MCC scores for all models. Examination of the t-SNE plot of responses in a Table 4. Results for selective coding after fine-tuning. Same experimental setup as before. Model Instruction F1 κ MCC Mixedbread large None 0.80 0 .53 0 .57 Nomic v1 None 0.78 0 .50 0 .52 J...

work page
[5]

RK Fussell, EM Stump, NG Holmes, Method to assess the trustworthiness of machine coding at scale. Phys. Rev. Phys. Educ. Res. 20, 010113 (2024)

work page 2024
[6]

A Arseniev-Koehler, SD Cochran, VM Mays, KW Chang, JG Foster, Integrating topic modeling and word embedding to characterize violent deaths. Proc. Natl. Acad. Sci . 119, e2108801119 (2022)

work page 2022
[7]

DM Blei, AY Ng, MI Jordan, Latent dirichlet allocation. J. Mach. Learn. Res . 3, 993–1022 (2003)

work page 2003
[8]

T Odden, A Marin, J Rudolph, How has science education changed over the last 100 years? an analysis using natural language processing. Sci. Educ. 105, 653–680 (2021)

work page 2021

Show all 36 references

[9]

R Bijker, SS Merkouris, NA Dowling, SN Rodda, Chatgpt for automated qualitative research: Content analysis. J. Med. Internet Res . 26, e59050 (2024)

2024
[10]

F Mumuni, A Mumuni, Explainable artificial intelligence (xai): from inherent explainability to large language models (2025)

2025
[11]

AY Cui, P Yu, Do language models have bayesian brains? distinguishing stochastic and deterministic decision patterns within large language models (2025)

2025
[12]

arXiv preprint arXiv:2408.05212 (2024) Discusses privacy threats and protections for human-subjects data in LLMs

M Miranda, et al., Preserving privacy in large language models: A survey on current threats and solutions. arXiv preprint arXiv:2408.05212 (2024) Discusses privacy threats and protections for human-subjects data in LLMs

2024 arXiv
[13]

N Jegham, M Abdelatti, L Elmoubarki, A Hendawi, How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference (2025)

2025
[14]

N Reimers, I Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks (2019)

2019
[15]

I Guyon, et al

A Vaswani, et al., Attention is all you need in Advances in Neural Information Processing Systems, eds. I Guyon, et al. (Curran Associates, Inc.), Vol. 30, (2017)

2017
[16]

A Merchant, E Rahimtoroghi, E Pavlick, I Tenney, What happens to BERT embeddings during fine-tuning? arXiv preprint arXiv:2004.14448 (2020)

2004 arXiv
[17]

N Muennighoff, N Tazi, L Magne, N Reimers, Mteb: Massive text embedding benchmark (2023)

2023
[18]

(Association for Computational Linguistics), pp

T Wolf, et al., Transformers: State-of-the-art natural language processing in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. (Association for Computational Linguistics), pp. 38–45 (2020)

2020
[19]

J Snell, K Swersky, RS Zemel, Prototypical networks for few-shot learning (2017)

2017
[20]

TOB Odden, H Tyseng, JT Mjaaland, MF Kreutzer, A Malthe-Sørenssen, Using text embeddings for deductive qualitative research at scale in physics education. Phys. Rev. Phys. Educ. Res. 20, 020151 (2024)

2024
[21]

3 edition, (2025) Online draft, accessed May 19, 2025

D Jurafsky, JH Martin, Vector semantics and embeddings in Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models . 3 edition, (2025) Online draft, accessed May 19, 2025

2025
[22]

EM Stump, M Hughes, G Passante, N Holmes, Comparing introductory and beyond-introductory students’ reasoning about uncertainty. Phys. Rev. Phys. Educ. Res . 19, 020147 (2023) Publisher: American Physical Society

2023
[23]

EM Stump, M Dew, G Passante, N Holmes, Context affects student thinking about sources of uncertainty in classical and quantum mechanics. Phys. Rev. Phys. Educ. Res . 19, 020157 (2023) Publisher: American Physical Society

2023
[24]

BM Zwickl, N Finkelstein, HJ Lewandowski, Incorporating learning goals about modeling into an upper-division physics laboratory experiment. Am. J. Phys. 82, 876–882 (2014). 7

2014
[25]

M G ¨unther, et al., Jina embeddings 2: 8192-token general-purpose text embeddings for long documents (2023)

2023
[26]

arXiv preprint arXiv:2409.10173 (2024)

S Sturua, et al., jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173 (2024)

2024 arXiv
[27]

S Lee, A Shakir, D Koenig, J Lipp, Open source strikes bread - new fluffy embeddings model (2024)

2024
[28]

arXiv preprint arXiv:2309.12871 (2023)

X Li, J Li, Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871 (2023)

2023 arXiv
[29]

arXiv preprint arXiv:2402.05672 (2024)

L Wang, et al., Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024)

2024 arXiv
[30]

Z Nussbaum, JX Morris, B Duderstadt, A Mulyar, Nomic embed: Training a reproducible long context text embedder (2024)

2024
[31]

H Su, et al., One embedder, any task: Instruction-finetuned text embeddings (2023)

2023
[32]

Biometrics 33, 159–174 (1977)

JR Landis, GG Koch, The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)

1977
[33]

L van der Maaten, G Hinton, Visualizing data using t-sne. J. Mach. Learn. Res . 9, 2579–2605 (2008)

2008
[34]

arXiv preprint arXiv:2406.01607 (2024)

H Cao, Recent advances in universal text embeddings: A comprehensive review of top-performing methods on the mteb benchmark. arXiv preprint arXiv:2406.01607 (2024)

2024 arXiv
[35]

(Association for Computational Linguistics), pp

T Gao, X Y ao, D Chen, Simcse: Simple contrastive learning of sentence embeddings in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). (Association for Computational Linguistics), pp. 6894–6910 (2021)

2021
[36]

JC Klie, B Webber, I Gurevych, Annotation error detection: Analyzing the past and present for a more coherent future. Comput. Linguist. 49, 157–198 (2023). 8

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

(Routledge, New Y ork), 2 edition, (2013)

SR Jones, V Torres, JL Arminio, Negotiating the Complexities of Qualitative Research in Higher Education: Fundamental Elements and Issues . (Routledge, New Y ork), 2 edition, (2013)

work page 2013

[2] [2]

X Zhai, Y Yin, JW Pellegrino, KC Haudek, LS and, Applying machine learning in science assessment: a systematic review. Stud. Sci. Educ . 56, 111–151 (2020)

work page 2020

[3] [3]

A Ratner, CD Sa, S Wu, D Selsam, C R ´e, Data programming: Creating large training sets, quickly (2017)

work page 2017

[4] [4]

Examination of the t-SNE plot of responses in a Table 4

The results show a substantial improvement between ten and twenty percent on both Kappa and MCC scores for all models. Examination of the t-SNE plot of responses in a Table 4. Results for selective coding after fine-tuning. Same experimental setup as before. Model Instruction F1 κ MCC Mixedbread large None 0.80 0 .53 0 .57 Nomic v1 None 0.78 0 .50 0 .52 J...

work page

[5] [5]

RK Fussell, EM Stump, NG Holmes, Method to assess the trustworthiness of machine coding at scale. Phys. Rev. Phys. Educ. Res. 20, 010113 (2024)

work page 2024

[6] [6]

A Arseniev-Koehler, SD Cochran, VM Mays, KW Chang, JG Foster, Integrating topic modeling and word embedding to characterize violent deaths. Proc. Natl. Acad. Sci . 119, e2108801119 (2022)

work page 2022

[7] [7]

DM Blei, AY Ng, MI Jordan, Latent dirichlet allocation. J. Mach. Learn. Res . 3, 993–1022 (2003)

work page 2003

[8] [8]

T Odden, A Marin, J Rudolph, How has science education changed over the last 100 years? an analysis using natural language processing. Sci. Educ. 105, 653–680 (2021)

work page 2021