Pith. sign in

REVIEW 3 major objections 5 minor 28 references

BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned DistilBERT classifier reproduces expert CO-PO alignment scores with 98.66% accuracy, making automated Course Articulation Matrices practical.

desk verdict A solid first application of BERT-family transfer learning to CO-PO alignment, but the headline 98.66% accuracy rests on an augmentation-before-split protocol that leaks paraphrases into validation. read the letter →

arxiv 2411.14254 v1 pith:MZYIJ3WM submitted 2024-11-21 cs.LG cs.AIcs.CY

Natenaile Asmamaw Shiferaw , Simpenzwe Honore Leandre , Aman Sinha , Dillip Rout This is my paper · ORCID
classification cs.LGcs.AIcs.CY
keywords coursearticulationmatrixCO-POalignmentBERTDistiltransferlearningLIMEexplainableAItextclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Accreditation bodies require engineering programs to map each course's learning outcomes to program outcomes with a numerical score from 0 to 3, a process faculty usually do by hand. This paper claims that a fine-tuned BERT-family model can automate that mapping: DistilBERT reaches 98.66% accuracy on the validation set, well above the best classical baseline of Random Forest at 87.93%. The authors argue this makes automated Course Articulation Matrix construction practical, and they add LIME explanations so each predicted score can be traced to specific words in the course and program descriptions. If the claim holds, institutions could generate CAMs from plain text quickly and with less subjective variation, though the robustness of that accuracy to genuinely unseen data is the key question.

What carries the argument

The load-bearing mechanism is a fine-tuned transformer language model, specifically a BERT-family encoder (BERT Base, DistilBERT, ALBERT, or RoBERTa) whose final classification layer is replaced by a four-way head that outputs an alignment score of 0, 1, 2, or 3 for an input CO-PO/PSO pair. Transfer learning via weight initialization lets a model pretrained on general text adapt to curriculum language, while a 30% synonym-replacement augmentation expands the 1,840 curated pairs to 8,949 balanced pairs and shuffling removes order bias. LIME, short for Local Interpretable Model-agnostic Explanations, is then applied after training to highlight which words in the CO or PO/PSO description pushed the prediction toward a given score.

What would settle it

Recompute the experiment with the train/validation split made on the 1,840 original pairs before any augmentation, so that no augmented paraphrase of a training pair appears in the validation set; if accuracy falls materially below 98.66%, the reported number does not establish generalization to unseen course or program texts.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fine-tuned BERT-family model can automate the construction of a Course Articulation Matrix by predicting, for each CO-PO or CO-PSO pair, the expert-assigned alignment score of 0, 1, 2, or 3. On a validation set of 1,790 augmented pairs built from 22 courses, DistilBERT achieves 98.66% accuracy, precision, recall, and F1-score, outperforming BERT-base at 98.32%, RoBERTa at 97.99%, ALBERT at 96.87%, and all three classical baselines. The paper also reports that DistilBERT's training time is the shortest among the transformers, and that LIME explanations align with the intended scores on four example pairs. The authors interpret these results as demonstrating that transfer learning with BERT-based models makes automated CAM generation both accurate and interpretable.

Load-bearing premise

The load-bearing premise is that the validation set measures true generalization: because the 30% synonym augmentation was applied before the 80/20 split, near-duplicate paraphrases of the same original CO-PO pair may appear in both training and validation, so the 98.66% accuracy could partly reflect recognition of paraphrases the model already saw rather than prediction on genuinely new texts.

Editorial extensions

If this is right

  • If the 98.66% validation accuracy holds on unseen course and program texts, institutions can generate the numerical CO-PO/PSO mapping cells of a CAM automatically, leaving faculty to review rather than draft the scores.
  • The augmentation-and-shuffle result pins the performance gain to data augmentation plus order randomization, not to the base model alone.
  • BERT-family transfer learning beats the best traditional classifier by roughly 11 points, so a pretrained language model is the load-bearing component of the pipeline.
  • LIME explanations tie each automated alignment score to specific words in the CO/PO text, giving educators a human-readable rationale for each cell of the matrix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If a held-out evaluation were run on original, non-augmented pairs only, I would expect the reported accuracy to drop, because the current split can place synonym paraphrases of the same underlying pair on both sides of the train/validation boundary.
  • A natural next experiment is to measure cross-institution generalization by training on one university's CAMs and validating on entire held-out courses from several other institutions, rather than a single showcase course.
  • The same pairwise text-classification setup could be reused for other curriculum-alignment tasks, such as mapping assessments to competencies or mapping program outcomes to accreditation criteria, where the four-point scale is replaced by the appropriate rubric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses automated construction of Course Articulation Matrices (CAMs) by classifying CO-PO/PSO pairs into alignment scores 0-3. The authors construct a manually curated dataset of 1,840 pairs from C.V. Raman Global University, apply 30% synonym-based data augmentation to obtain 8,949 pairs, and then fine-tune BERT-base, DistilBERT, ALBERT, and RoBERTa, comparing them against Decision Tree, Random Forest, and XGBoost baselines. They report that DistilBERT achieves 98.66% accuracy, precision, recall, and F1-score on the validation set, and they use LIME to provide local explanations for predictions. The central claim is that transfer learning with BERT-based models can automate CAM construction at near-perfect accuracy with interpretability.

Significance. If the reported performance were obtained under an honest evaluation protocol, the paper would offer a practically useful tool for accreditation-related curriculum mapping, a task that is currently manual and subjective. The strengths of the submission include a publicly available curated dataset, a systematic comparison of four transformer models and three classical baselines, and an interpretability analysis with LIME. The external source of the ground-truth labels means the supervised task itself is not circular. However, the headline performance number is compromised by the experimental protocol, as detailed below, so the significance of the empirical contribution cannot be assessed from the current results.

major comments (3)
  1. [§3.1.2 and §3.3.3] Data augmentation is applied to all 1,840 original CO-PO/PSO pairs before the 80/20 train/validation split. Since each original pair generates roughly five near-duplicate paraphrases, a random split almost guarantees that near-duplicates of the same original pair appear in both the training and validation sets. The validation set is therefore not an independent sample of unseen outcome text, and the reported 98.66% accuracy may reflect recognition of training-set paraphrases rather than generalization to new pairs. The authors should split the original 1,840 pairs into training and validation first and apply augmentation only to the training portion, or use a grouped split that keeps all paraphrases of the same original pair in the same fold.
  2. [§4.4, Table 6] The ablation shows accuracy jumping from 83.70% pre-augmentation to 98.66% with augmentation and shuffling. This dramatic increase is exactly the pattern expected from the leakage described in the previous comment, because the model can memorize paraphrase variants that also appear in validation. The central performance claim is therefore unsupported as reported. The authors need to provide validation metrics under a leakage-free protocol, for example by splitting before augmentation or by deduplicating validation pairs, and report whether the near-perfect accuracy persists.
  3. [§4.5.2, Figure 8] The CBIT test case is presented as evidence of cross-institutional generalization, but it is a single anecdotal course with one misclassified cell and no aggregate metrics over the full CBIT dataset. A randomly selected course does not constitute a held-out evaluation, especially without a description of how the CBIT data were obtained, labeled, or compared against the model's predictions. The claim that the model 'performs well even when applied to data from an institution not included in the training and validation sets' should be either supported by a quantitative evaluation on a held-out institution or substantially weakened.
minor comments (5)
  1. [§3.3.2, Eq. (6)] The categorical cross-entropy formula uses y_ij, but the surrounding text says y_ij takes values 0, 1, 2, or 3. In a one-hot encoding for cross-entropy, y_ij should be 0 or 1; please clarify the notation.
  2. [§3.3.1 and §3.3.3] Section 3.3.1 describes the Adam optimizer, while Section 3.3.3 states that AdamW was used for training. Please clarify which optimizer was actually employed, since this affects reproducibility.
  3. [§3.1.1 and §4.4] Section 3.1.1 says only the augmented version was used for training and evaluation, but Table 6 includes a pre-augmentation baseline. Please clarify that the pre-augmentation experiments used the original 1,840 pairs and that the split protocol was consistent across all settings.
  4. [§4.6, Figure 9] The LIME explanation colors (orange and green) are mentioned in the text but not defined in the figure caption. Please add a caption note explaining what each color indicates.
  5. [§3.1.2] The synonym augmentation procedure is not fully specified: it does not state which thesaurus or WordNet version was used, how random selection was seeded, or whether augmentation was applied to both the CO and the PO/PSO text in each pair. These details are needed for reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the supervision labels are external and the fine-tuning pipeline is standard; the only circularity-adjacent issue is augmentation-before-split, which makes validation partially self-referential and is best treated as a soundness flaw.

full rationale

The paper's claimed derivation chain does not reduce to its inputs by construction. The alignment labels in Section 3.1.1 are "sourced from the university's pre-curated CAM," so the supervised signal is external to the model. No parameter is fitted to the validation labels; the validation set is used only for evaluation. The BERT fine-tuning protocol (Section 3.3.3) is ordinary transfer learning with a modified classification head, and the reported 98.66% accuracy is an empirical result, not an identity. The LIME analysis (Section 4.6) is post-hoc interpretability and does not enter the training objective. The only circularity-adjacent concern is that synonym augmentation (Section 3.1.2) is performed before the 80/20 train/validation split (Section 3.3.3), so augmented paraphrases of the same original CO-PO/PSO pair can appear in both sets; this makes the validation accuracy partially self-referential and threatens the generalization claim. That is a data-splitting or measurement flaw, however, not a derivation-level equivalence: the model is not defined in terms of the validation labels, and no predicted quantity is identical to its training input by construction. Since no specific reduction can be exhibited, no circular step is recorded; the score of 1 registers the augmentation/validation leakage as a minor self-referentiality concern rather than a circular derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on four hand-chosen modeling choices (augmentation ratio, learning rate, epochs, baseline hyperparameters) and three domain assumptions about label quality, label preservation under paraphrase, and validation-set independence. No new entities are introduced. The most consequential entry is the validation-set independence assumption: augmentation before the split likely inflates the headline accuracy.

free parameters (4)
  • synonym replacement ratio = 30%
    Hand-chosen fraction of words replaced during augmentation; directly drives the reported accuracy gain from 83.70% to 98.66% and is not ablated.
  • learning rate = 5e-5
    Standard BERT fine-tuning choice; effect on the headline accuracy is not tested (Section 3.3.1).
  • training epochs = 10
    Selected by experimentation; no early stopping or epoch sensitivity analysis is reported (Section 3.3.3).
  • classical ML hyperparameters (tree depth, estimator counts) = max_depth 10 and 30, estimators 200
    Arbitrary-looking settings for Decision Tree and Random Forest baselines; no tuning procedure is specified (Section 3.3.3).
assumptions (4)
  • domain assumption Expert pre-curated CAM labels from one university are correct ground truth for alignment scores.
    Scores from C.V. Raman Global University's pre-curated CAM are treated as error-free labels; no inter-annotator agreement check or external audit is reported (Section 3.1.1).
  • domain assumption Synonym substitution preserves the alignment score label.
    Replacing 30% of words with synonyms is assumed to keep the CO-PO semantic relation intact; no human or automated verification of label preservation is provided (Section 3.1.2).
  • ad hoc to paper The train/validation split after augmentation yields an independent validation set.
    Augmentation precedes the 80/20 split, so paraphrases of the same original pair can fall on both sides; the paper relies on this protocol to claim validation accuracy (Sections 3.1.2 and 3.3.3).
  • domain assumption Validation accuracy is a proxy for accuracy on other institutions' CO/PO phrasings.
    One test course from CBIT is used to assert strong generalization; the assumption that phrasing diversity across institutions is captured by synonym augmentation is unstated and unmeasured (Section 4.5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI." pith.science (2026). https://pith.science/paper/MZYIJ3WM

@misc{pith2026241114254,
  author       = {Pith},
  title        = {Pith review of: BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZYIJ3WM}},
  note         = {Machine review of arXiv:2411.14254}
}
read the original abstract

Course Outcome (CO) and Program Outcome (PO)/Program-Specific Outcome (PSO) alignment is a crucial task for ensuring curriculum coherence and assessing educational effectiveness. The construction of a Course Articulation Matrix (CAM), which quantifies the relationship between COs and POs/PSOs, typically involves assigning numerical values (0, 1, 2, 3) to represent the degree of alignment. In this study, We experiment with four models from the BERT family: BERT Base, DistilBERT, ALBERT, and RoBERTa, and use multiclass classification to assess the alignment between CO and PO/PSO pairs. We first evaluate traditional machine learning classifiers, such as Decision Tree, Random Forest, and XGBoost, and then apply transfer learning to evaluate the performance of the pretrained BERT models. To enhance model interpretability, we apply Explainable AI technique, specifically Local Interpretable Model-agnostic Explanations (LIME), to provide transparency into the decision-making process. Our system achieves accuracy, precision, recall, and F1-score values of 98.66%, 98.67%, 98.66%, and 98.66%, respectively. This work demonstrates the potential of utilizing transfer learning with BERT-based models for the automated generation of CAMs, offering high performance and interpretability in educational outcome assessment.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 18 canonical work pages

  1. [1]

    Journal of Engineering Education Transformations 35(4), 104–111 (2022) https://doi.org/10.16920/jeet/2022/v35i4/22109 22

    Lavanya, C., Murthy, J.N.: Assessment and attainment of course outcomes and program outcomes. Journal of Engineering Education Transformations 35(4), 104–111 (2022) https://doi.org/10.16920/jeet/2022/v35i4/22109 22

    work page doi:10.16920/jeet/2022/v35i4/22109 2022

  2. [2]

    Journal of Engi- neering Education Transformations (2016) https://doi.org/10.16920/jeet/2016/ v0i0/85662

    Admuthe, L., Loni, D.: Course outcome-program outcome mapping matrix & attainment-issues and model based solutions for tier ii category. Journal of Engi- neering Education Transformations (2016) https://doi.org/10.16920/jeet/2016/ v0i0/85662

    work page doi:10.16920/jeet/2016/ 2016

  3. [3]

    191–198 (2022)

    Mundhe, S., Taralkar, S.: Effective Means, Methods, And Performance Evaluation for Conducting Practical Sessions for Improved Learning Outcomes, pp. 191–198 (2022). https://doi.org/10.4018/978-1-6684-4210-4.ch010

    work page doi:10.4018/978-1-6684-4210-4.ch010 2022

  4. [4]

    Journal of Engineering Education Transformations 38, 85–95 (2024) https://doi.org/10.16920/jeet/2024/v38i2/24192

    Yadav, P., Tomar, A., Soni, R., Dubey, G., Bhargava, C., Kumari, J.: Metrics with performance indicators used for calculating the mapping strength of course out- come with program outcome. Journal of Engineering Education Transformations 38, 85–95 (2024) https://doi.org/10.16920/jeet/2024/v38i2/24192

    work page doi:10.16920/jeet/2024/v38i2/24192 2024

  5. [5]

    European Journal of Engineering Education 46 (2020) https://doi.org/10.1080/03043797.2020.1852533

    Liew, C., Puteh, M., Mohammad, S., Kiew, P.L.: Review of engineering pro- gramme outcome assessment models. European Journal of Engineering Education 46 (2020) https://doi.org/10.1080/03043797.2020.1852533

    work page arXiv 2020

  6. [6]

    44–48 (2023)

    Sengupta, S., Das, A.: Automated mapping of course outcomes to program out- comes using natural language processing and machine learning, pp. 44–48 (2023). https://doi.org/10.1109/ASPCON59071.2023.10396272

    work page arXiv 2023

  7. [7]

    American journal of pharmaceutical education 71, 20 (2007)

    Plaza, C., Draugalis, J., Slack, M., Skrepnek, G., Sauer, K.: Curriculum map- ping in program assessment and evaluation. American journal of pharmaceutical education 71, 20 (2007)

    work page 2007

  8. [8]

    Higher Education Research & Development31, 217–231 (2012) https: //doi.org/10.1080/07294360.2011.554387

    Spencer, D., Riddle, M., Knewstubb, B.: Curriculum mapping to embed graduate capabilities. Higher Education Research & Development31, 217–231 (2012) https: //doi.org/10.1080/07294360.2011.554387

    work page arXiv 2012

Show all 28 references
  1. [9]

    137–154 (2017)

    Lindn, J., Annala, J., Coate, K.: The Role of Curriculum Theory in Contemporary Higher Education Research and Practice, pp. 137–154 (2017). https://doi.org/10. 1108/S2056-375220170000003008

    2017

  2. [10]

    African Journal of Health Professions Education 11(1), 27–31 (2019)

    Treadwell, I., Ahlers, O., Botha, G.: Initiating curriculum mapping on the web-based, interactive learning opportunities, objectives and outcome platform (looop). African Journal of Health Professions Education 11(1), 27–31 (2019)

    2019

  3. [11]

    Focus on Health Professional Education: A Multi-Professional Journal 21(1), 91–113 (2020) https://doi.org/10.11157/fohpe

    Watson, E.G.S., Steketee, C., Mansfield, K., Moore, M., Dalziel, B., Damodaran, A., Walker, B., Duvivier, R.J., Hu, W.: Curriculum mapping for health pro- fessions education: a typology. Focus on Health Professional Education: A Multi-Professional Journal 21(1), 91–113 (2020) ...

    2020 doi

  4. [12]

    23 International Journal of Advanced Computer Science and Applications 11, 747– 754 (2020) https://doi.org/10.14569/IJACSA.2020.0111285

    Alshanqiti, A., Alam, T., Benaida, M., Namoun, A., Taleb, A.: A rule-based approach toward automating the assessments of academic curriculum mapping. 23 International Journal of Advanced Computer Science and Applications 11, 747– 754 (2020) https://doi.org/10.14569/IJACSA.2020.0111285

    2020

  5. [13]

    McGraw- Hill, New York (1983)

    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw- Hill, New York (1983)

    1983

  6. [14]

    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis, vol. 41, pp. 391–407 (1990)

    1990

  7. [15]

    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation, vol. 3, pp. 993– 1022 (2003)

    2003

  8. [16]

    Nature 323, 533–536 (1986)

    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by backpropagating errors. Nature 323, 533–536 (1986)

    1986

  9. [17]

    Neural Computation 9(8), 1735–1780 (1997) https://doi.org/10.1162/neco.1997.9.8.1735

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997) https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  10. [18]

    https://arxiv.org/ abs/1706.03762

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need (2023). https://arxiv.org/ abs/1706.03762

    2023 arXiv

  11. [19]

    https://arxiv

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019). https://arxiv. org/abs/1810.04805

    2019 arXiv

  12. [20]

    https://arxiv.org/abs/1910

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2020). https://arxiv.org/abs/1910. 01108

    2020

  13. [21]

    https://arxiv.org/abs/1909.11942

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2020). https://arxiv.org/abs/1909.11942

    2020 arXiv

  14. [22]

    https://arxiv.org/abs/1907.11692

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). https://arxiv.org/abs/1907.11692

    2019 arXiv

  15. [23]

    Wadsworth International Group, Belmont, CA (1986)

    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth International Group, Belmont, CA (1986)

    1986

  16. [24]

    Machine Learning 45(1), 5–32 (2001) https://doi

    Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001) https://doi. org/10.1023/A:1010933404324

    2001 doi

  17. [26]

    Knowledge and Data Engineer- ing, IEEE Transactions on 22, 1345–1359 (2010) https://doi.org/10.1109/TKDE

    Pan, S., Yang, Q.: A survey on transfer learning. Knowledge and Data Engineer- ing, IEEE Transactions on 22, 1345–1359 (2010) https://doi.org/10.1109/TKDE. 2009.191

    2010 doi

  18. [27]

    https: //arxiv.org/abs/1412.6980

    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (2017). https: //arxiv.org/abs/1412.6980

    2017 arXiv

  19. [28]

    MIT Press, Cambridge, MA (2016)

    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge, MA (2016). http://www.deeplearningbook.org

    2016

  20. [29]

    why should i trust you?

    Ribeiro, M.T., Singh, S., Guestrin, C.: “why should i trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining (KDD ’16), pp. 1135–1144. Association for Computing Machine...

    2016

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.