An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education

Ibrahim Demir; Ramteja Sajja; Yusuf Sermet

REVIEW 3 major objections 4 minor 7 references

An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small open-source embedding model, fine-tuned with both ranking and similarity losses on 3,197 synthetic syllabus question-answer pairs, matches or approaches proprietary OpenAI embeddings on course, faculty, and TA retrieval while…

desk verdict A useful open-source contribution with a weak evaluation: the headline claim rests on end-to-end QA accuracy rather than direct retrieval, and the text contains a factual error. read the letter →

arxiv 2505.04916 v1 pith:EI3NUMR4 submitted 2025-05-08 cs.CL cs.AIcs.IR

Ramteja Sajja , Yusuf Sermet , Ibrahim Demir This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords semanticretrievaldomain-specificembeddingsdual-lossfine-tuningMultipleNegativesRankingLossCosineSimilaritysyllabusquestionansweringretrieval-augmentedgenerationeducationalNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a small, open-source sentence embedding model can be made competitive with proprietary commercial embeddings for retrieving information from university course syllabi, by fine-tuning it on a small synthetic dataset of educational question-answer pairs. The central claim is that a dual-loss objective, combining MultipleNegativesRankingLoss with CosineSimilarityLoss, yields embeddings that beat every tested open-source baseline and approach OpenAI's text-embedding-3 models on course, faculty, and TA queries. The authors construct a 3,197-pair synthetic dataset spanning synonyms, paraphrases, and implicit-to-explicit mappings, and evaluate through a chunk-based retrieval pipeline followed by a lightweight generator with manual verification. If right, the result matters because institutions could get near-proprietary retrieval quality with transparent, low-cost, locally hosted models.

What carries the argument

The load-bearing mechanism is the dual-loss training objective. MultipleNegativesRankingLoss pulls each positive question-answer pair together while treating all other batch items as implicit negatives, giving relative ranking; CosineSimilarityLoss adds direct supervision so that similarity scores align with binary semantic labels (1 for similar, 0 for dissimilar). Two separate data loaders feed these losses, and the model, built on the all-MiniLM-L6-v2 architecture, is trained for 25 epochs on a 3,197-pair synthetic dataset. The combination is what the paper credits for handling synonymy, paraphrasing, and implicit phrasing in syllabus text.

What would settle it

Re-run the same 28-syllabus benchmark but score retrieval directly, marking a retrieval as correct only if the chunk containing the answer appears in the top 3, and drop the generator step. If the dual-loss model no longer beats the open-source baselines, or the gap to OpenAI grows, the paper's central claim about retrieval quality would not survive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that domain-specific fine-tuning of a compact open-source model, using both a ranking loss and an explicit cosine-similarity loss, produces a semantic retriever that outperforms all tested open-source embeddings (all-MiniLM-L6-v2, multi-qa-MiniLM-L6-cos-v1, msmarco-distilbert-base-v4, nli-roberta-base-v2) across all three syllabus question categories and narrows the gap with proprietary OpenAI embeddings. In the Teaching Assistant category, the dual-loss model reaches 87.50% accuracy versus 90.18% for text-embedding-3-large, and in Faculty Information it reaches 88.10%, close to text-embedding-3-small (89.29%) and text-embedding-ada-002 (91.67%). The dual-loss model also matches or exceeds the MNRL-only model on every category, with the largest gain in Faculty Information (88.10% vs. 84.52%).

Load-bearing premise

The results treat end-to-end answer accuracy, where a generator reads the top-3 retrieved chunks and a human judges the answer, as a measure of retrieval quality; if the generator can answer from its own knowledge or fail despite good retrieval, the reported differences would not cleanly reflect embedding performance.

Editorial extensions

If this is right

Both fine-tuned models beat the generic open-source baselines, so domain-specific fine-tuning on a small synthetic corpus is a viable route to better educational retrieval.
The dual-loss model consistently matches or exceeds the MNRL-only model, so adding similarity calibration appears to help on fine-grained distinctions such as TA versus faculty names.
The performance gap to proprietary models shrinks to a few percentage points, supporting the feasibility of open, self-hosted alternatives for course chatbots and RAG pipelines.
Treating "Sorry, I don't know" as a valid answer when content is absent shows that retrieval pipelines can be evaluated for hallucination avoidance, not just raw hit rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A retrieval-only evaluation that checks whether the correct chunk appears in the top 3, without any generator, would separate embedding quality from the answer-generation step; the paper's current numbers mix the two.
The dual-loss recipe could transfer to other structured academic documents (lecture notes, policies, program descriptions) with comparably small synthetic datasets, since the paper's corpus is only 3,197 pairs.
Because the synthetic negative set is small (487 pairs), mining harder negatives from real syllabi would likely stress-test whether the calibration gain persists under more challenging distractors.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: the paper is a modest but genuine contribution—two open-source fine-tuned embedding models for educational QA, trained on a synthetic dataset and released on Hugging Face, plus a training-data repo. That is real value, and the central direction (domain-specific fine-tuning of a small model for syllabus retrieval) is sensible. But the headline comparison against proprietary embeddings is not actually supported by the evidence as presented, because the evaluation measures end-to-end QA accuracy, not retrieval quality.

What's new and good: the fine-tuned models themselves, the released synthetic dataset, and the use of real syllabi for evaluation. The dual-loss setup is a known recipe, but the application to higher-education syllabi and the specific released artifacts are new. The paper is clearly written and honest about several limitations (chunking noise, boilerplate, future work). It earns credit for shipping code and weights.

Where it's soft: The benchmark passes the top-3 retrieved chunks to GPT-4o-mini and manually judges the answer. That conflates retrieval with the generator's parametric knowledge and the judge's effort. The Course Information category is particularly weak: when a syllabus lacks credit hours, an answer of "Sorry, I don't know" is counted as valid, so a model that retrieves irrelevant text can still score full credit whenever the generator declines to answer. This explains the 100% saturation across top models and strips that category of discriminative power. Elsewhere the margins are thin—88.10% vs 85.71% on Faculty is two or three answers out of 84—and there are no error bars or significance tests. The paper also contains a clear internal contradiction: it says the dual-loss model "surpasses" text-embedding-3-small (89.29%) and ada-002 (91.67%) while reporting 88.10% for its own model, which is lower. That typo undermines confidence in the written claims. Finally, the evaluation syllabi and scripts are not released, only the training data and model weights, so exact replication is limited.

None of this kills the paper. Direct retrieval metrics (Recall@k, MRR, nDCG) would likely show the fine-tuned models are indeed better than off-the-shelf MiniLM for this domain—the QA numbers point that way, even if they don't prove it. But right now the paper is an engineering report with a promising but under-analyzed experiment.

Recommendation: this deserves serious peer review, not a desk rejection. The artifacts are reusable and the claim is testable. A good referee should ask for a direct retrieval evaluation, error bars or bootstrap confidence intervals, a fixed Course Information protocol, and correction of the numeric contradiction in Section 3.2.

Referee Report

3 major / 4 minor

Summary. The paper introduces two open-source embedding models fine-tuned for semantic retrieval over university course syllabi, built on all-MiniLM-L6-v2. A synthetic training set of 3,197 sentence pairs was created via manual curation and LLM-assisted generation, covering synonymy, paraphrases, and implicit-explicit mappings. One model uses MultipleNegativesRankingLoss only; the second combines MNRL with CosineSimilarityLoss. Evaluation is performed on 28 real syllabi with fixed questions in three categories (Course, Faculty, Teaching Assistant Information). For each question, the top-3 chunks retrieved by cosine similarity are passed to GPT-4o-mini, and the generated answer is manually judged against the syllabus. The authors report that both fine-tuned models outperform open-source baselines and that the dual-loss model approaches proprietary OpenAI embeddings, with a stated claim of surpassing some OpenAI models in the Faculty Information category. The paper also discusses limitations of chunking and boilerplate content and releases the dataset and model weights.

Significance. If the central claim holds, the contribution is a transparent, reproducible framework for domain-specific embedding fine-tuning in education, with publicly released models and data. The paper's strengths include the open release of the training dataset and model weights, the use of real syllabi for evaluation, and a clear description of the synthetic data construction and dual-loss training procedure. The domain (course-syllabus retrieval) is practical and underexplored, and the comparison against both open-source and proprietary baselines is useful. However, the significance is limited by the evaluation design: the headline metric is end-to-end QA accuracy obtained through a generator and manual judgment, not a direct measure of retrieval quality. The evaluation corpus is small (28 syllabi), the benchmark is author-constructed, and the reported differences between models are often small in absolute count. The paper's contribution would be strengthened substantially by adding standard retrieval metrics (e.g., Recall@k, MRR, nDCG) and statistical uncertainty estimates.

major comments (3)

[Sections 3.1.1 and 3.1.3; Table 3]
[Section 3.2, Course Information category]
[Section 3.2, Faculty Information paragraph]

minor comments (4)

[Table 1]
[Section 2.4]
[Table 2 and throughout]
[Section 3.3.1]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is an empirical evaluation on held-out syllabi and does not reduce to its training inputs.

full rationale

The derivation chain is linear and self-contained. The models are fine-tuned on a synthetic corpus (Section 2.2) and evaluated on 28 real university syllabi with a fixed question set and external baselines (Section 3.1). No parameter is fitted to the evaluation set; the dual-loss objective trains on labeled pairs, while retrieval accuracy is computed on held-out syllabus documents. The authors' prior work (e.g., Sajja et al. 2024; Sajja et al. 2023a) appears in motivational framing and is not used to justify the empirical result. The evaluation pipeline (top-3 retrieval followed by GPT-4o-mini generation and manual validation) is a proxy for retrieval quality and could conflate retrieval with generation, but that is a validity limitation rather than a circular construction. Similarly, the treatment of absent-information answers as valid in the Course Information category (Section 3.2) can saturate that category, but it applies to all models and does not force the headline ranking by definition. The overlap between the training taxonomy and evaluation categories is expected in domain adaptation and does not make the reported accuracies equivalent to the input by construction.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted theory parameters. It relies on standard ML hyperparameters (learning rates, epochs, batch size) chosen without ablations, plus domain assumptions about the synthetic data and the generative evaluation pipeline. The central claim is an empirical result, not a derivation.

free parameters (9)

Learning rate (MNRL model) = 2e-5
Chosen by hand for training; no ablation or tuning curve reported.
Learning rate (dual-loss model) = 1e-5
Set lower for the dual objective; no comparison across learning rates.
Number of epochs = 25
Arbitrary; no early stopping or validation set is mentioned.
Batch size = 64
Training batch size for both DataLoaders.
Warmup fraction = 15% (MNRL), 10% (dual)
Chosen scheduler parameters with no sensitivity analysis.
Weight decay = 0.01
Regularization hyperparameter, not optimized.
Chunk size = ~300 words
Fixed-size segmentation for evaluation; the authors acknowledge this as a limitation.
Top-k retrieved chunks = 3
Number of chunks concatenated for answer generation; no variation tested.
Dual-loss combination weights
The paper does not specify how MNRL and CosineSimilarityLoss are weighted or alternated, a reproducibility gap.

assumptions (4)

domain assumption Cosine similarity in embedding space reflects semantic relatedness for syllabus retrieval.
Used as the retrieval scoring function; standard in the field but not validated for this corpus.
domain assumption The synthetic pairs generated with GPT-4 and manual curation are representative of real student queries over syllabi.
Dataset design in Section 2.2; the evaluation categories mirror the training categories, which may inflate the apparent benefit.
domain assumption GPT-4o-mini generates factually reliable answers when given relevant chunks, and manual grading correctly identifies factual support.
Used in Sections 3.1.1 and 3.1.3; no inter-annotator agreement or generator error analysis is reported.
domain assumption Fine-tuning on 3,197 pairs transfers to unseen syllabi better than the base model.
The core empirical claim; relies on standard supervised learning assumptions about distribution shift.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education." pith.science (2026). https://pith.science/paper/EI3NUMR4

@misc{pith2026250504916,
  author       = {Pith},
  title        = {Pith review of: An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EI3NUMR4}},
  note         = {Machine review of arXiv:2505.04916}
}

read the original abstract

Recent advances in AI have catalyzed the adoption of intelligent educational tools, yet many semantic retrieval systems remain ill-suited to the unique linguistic and structural characteristics of academic content. This study presents two open-source embedding models fine-tuned for educational question answering, particularly in the context of course syllabi. A synthetic dataset of 3,197 sentence pairs, spanning synonymous terminology, paraphrased questions, and implicit-explicit mappings, was constructed through a combination of manual curation and large language model (LLM)-assisted generation. Two training strategies were evaluated: (1) a baseline model fine-tuned using MultipleNegativesRankingLoss (MNRL), and (2) a dual-loss model that combines MNRL with CosineSimilarityLoss to improve both semantic ranking and similarity calibration. Evaluations were conducted on 28 university course syllabi using a fixed set of natural language questions categorized into course, faculty, and teaching assistant information. Results demonstrate that both fine-tuned models outperform strong open-source baselines, including all-MiniLM-L6-v2 and multi-qa-MiniLM-L6-cos-v1, and that the dual-loss model narrows the performance gap with high-performing proprietary embeddings such as OpenAI's text-embedding-3 series. This work contributes reusable, domain-aligned embedding models and provides a replicable framework for educational semantic retrieval, supporting downstream applications such as academic chatbots, retrieval-augmented generation (RAG) systems, and learning management system (LMS) integrations.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages

[1]

Alsultan, A., & Razak, F. A. (2024). Limitations of Language Models in The Oil & Gas Upstream Operations. All Days. https://doi.org/10.2523/iptc-23649-ea Bilad, M. R., Yaqin, L. N., & Zubaidah, S. (2023). Recent progress in the use of artificial intelligence tools in education. Jurnal Penelitian Dan Pengkajian Ilmu Pendidikan e -Saintika, 7(3), 279–315. h...

work page doi:10.2523/iptc-23649-ea 2024
[2]

Choudhary, S., Jaiswal, H

https://doi.org/10.62486/latia20232. Choudhary, S., Jaiswal, H. K., Rajeevan, A. S., Mukherjee, P. D., & Hudnurkar, S. (2024, October). Semantic Search of Educational Texts. In 2024 International Conference on Intelligent Systems and Advanced Applications (ICISAA) (pp. 1-5). IEEE. Cruciani, F., Moore, S., & Nugent, C. D. (2023, April). Comparing general p...

work page doi:10.62486/latia20232 2024
[42]

Sajja, R., Sermet, Y., Cwiertny, D., & Demir, I. (2023b). Integrating AI and learning analytics for data-driven pedagogical decisions and personalized interventions in education. arXiv preprint arXiv:2312.09548. Samuel, D. J., Sermet, M. Y., Mount, J., Vald, G., Cwiertny, D., & Demir, I. (2024). Application of Large Language Models in Developing Conversat...

arXiv 2023
[57]

B., & McClatchey, R

Shamdasani, J., Bloodsworth, P., Munir, K., Rahmouni, H. B., & McClatchey, R. (2011). MedMatch – towards domain specific semantic matching. In Lecture notes in computer science (pp. 375–382). https://doi.org/10.1007/978-3-642-22688-5_33 Shen, J., Tenenholtz, N., Hall, J. B., Alvarez-Melis, D., & Fusi, N. (2024). Tag-LLM: Repurposing General-Purpose LLMs f...

arXiv doi:10.48550/arxiv.2402.05140 2011
[81]

& Jamnik, M

https://doi.org/10.62802/3fr4f412 Espinosa Zarlenga, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., ... & Jamnik, M. (2022). Concept embedding models: Beyond the accuracy-explainability trade-off. Advances in Neural Information Processing Systems, 35, 21400-21413. Greene, R., Sanders, T., Weng, L., & Neelakantan, A. (2022, Decemb...

work page arXiv 2022
[148]

Kumar, A., Singh, D., & Vohra, R. (2023). Improving learning abilities using AI-Based education systems. In Advances in educational technologies and instructional design book series (pp. 137–155). https://doi.org/10.4018/979-8-3693-0378-8.ch006 Manchanda, J., Boettcher, L., Westphalen, M., & Jasser, J. (2024). The open source advantage in Large Language m...

work page doi:10.48550/arxiv.2412.12004 2023
[7056]

https://doi.org/10.31223/X5XT4K Sermet, Y., & Demir, I. (2021). A semantic web framework for automated smart assistants: A case study for public health. Big Data and Cognitive Computing, 5(4),

work page doi:10.31223/x5xt4k 2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Alsultan, A., & Razak, F. A. (2024). Limitations of Language Models in The Oil & Gas Upstream Operations. All Days. https://doi.org/10.2523/iptc-23649-ea Bilad, M. R., Yaqin, L. N., & Zubaidah, S. (2023). Recent progress in the use of artificial intelligence tools in education. Jurnal Penelitian Dan Pengkajian Ilmu Pendidikan e -Saintika, 7(3), 279–315. h...

work page doi:10.2523/iptc-23649-ea 2024

[2] [2]

Choudhary, S., Jaiswal, H

https://doi.org/10.62486/latia20232. Choudhary, S., Jaiswal, H. K., Rajeevan, A. S., Mukherjee, P. D., & Hudnurkar, S. (2024, October). Semantic Search of Educational Texts. In 2024 International Conference on Intelligent Systems and Advanced Applications (ICISAA) (pp. 1-5). IEEE. Cruciani, F., Moore, S., & Nugent, C. D. (2023, April). Comparing general p...

work page doi:10.62486/latia20232 2024

[3] [42]

Sajja, R., Sermet, Y., Cwiertny, D., & Demir, I. (2023b). Integrating AI and learning analytics for data-driven pedagogical decisions and personalized interventions in education. arXiv preprint arXiv:2312.09548. Samuel, D. J., Sermet, M. Y., Mount, J., Vald, G., Cwiertny, D., & Demir, I. (2024). Application of Large Language Models in Developing Conversat...

arXiv 2023

[4] [57]

B., & McClatchey, R

Shamdasani, J., Bloodsworth, P., Munir, K., Rahmouni, H. B., & McClatchey, R. (2011). MedMatch – towards domain specific semantic matching. In Lecture notes in computer science (pp. 375–382). https://doi.org/10.1007/978-3-642-22688-5_33 Shen, J., Tenenholtz, N., Hall, J. B., Alvarez-Melis, D., & Fusi, N. (2024). Tag-LLM: Repurposing General-Purpose LLMs f...

arXiv doi:10.48550/arxiv.2402.05140 2011

[5] [81]

& Jamnik, M

https://doi.org/10.62802/3fr4f412 Espinosa Zarlenga, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., ... & Jamnik, M. (2022). Concept embedding models: Beyond the accuracy-explainability trade-off. Advances in Neural Information Processing Systems, 35, 21400-21413. Greene, R., Sanders, T., Weng, L., & Neelakantan, A. (2022, Decemb...

work page arXiv 2022

[6] [148]

Kumar, A., Singh, D., & Vohra, R. (2023). Improving learning abilities using AI-Based education systems. In Advances in educational technologies and instructional design book series (pp. 137–155). https://doi.org/10.4018/979-8-3693-0378-8.ch006 Manchanda, J., Boettcher, L., Westphalen, M., & Jasser, J. (2024). The open source advantage in Large Language m...

work page doi:10.48550/arxiv.2412.12004 2023

[7] [7056]

https://doi.org/10.31223/X5XT4K Sermet, Y., & Demir, I. (2021). A semantic web framework for automated smart assistants: A case study for public health. Big Data and Cognitive Computing, 5(4),

work page doi:10.31223/x5xt4k 2021