pith. sign in

arxiv: 2606.00200 · v1 · pith:2VORPXYJnew · submitted 2026-05-29 · 💻 cs.CY

Student Competency Assessment and Presentation Methods Based on Algorithm Courses

Yingqi Zhang , Ninghan Zheng , Shanshan Li , Weidong Liu This is my paper

Pith reviewed 2026-06-28 19:41 UTC · model grok-4.3

classification 💻 cs.CY
keywords competency assessmentalgorithm coursesMarkov process modelingxAPI specificationCC2020 modelstudent clusteringpersonalized teachingcurriculum optimization
0
0 comments X

The pith

Adapting 169 students' programming data to xAPI format and applying Markov modeling quantifies competencies and identifies clusters in algorithm courses.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops methods to assess student competencies in algorithm courses using the CC2020 model that combines knowledge, skills, and dispositions. Data from programming experiments and written assignments of 169 students is standardized to the xAPI specification for consistent analysis. Markov process modeling tracks sequences of student behaviors to uncover cognitive patterns, while other techniques measure the three competency areas and group students into clusters. Course experiment difficulty is assessed through proactiveness metrics based on submission timeliness. The outcome is a framework that supports tailored teaching and curriculum adjustments according to the observed clusters.

Core claim

By adapting programming experiment and written assignment data from 169 students to the xAPI specification and employing Markov process modeling on behavioral sequences, competencies in knowledge, skills, and dispositions can be quantified, distinct student clusters identified, and course experiment difficulty measured via proactiveness metrics, yielding a scalable assessment framework for personalized teaching and curriculum optimization in algorithm courses.

What carries the argument

Markov process modeling of behavioral sequences after xAPI adaptation of student data, used to quantify competencies and form clusters.

If this is right

  • Instructors can design interventions matched to each identified student cluster.
  • Experiment difficulty levels can be adjusted using the proactiveness metrics from submission data.
  • The framework can incorporate additional student records to refine and validate the competency clusters.
  • The same methods provide a starting point for applying competency assessment across other computer science courses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Cluster-based grouping could be tested by measuring whether targeted interventions produce better learning gains than uniform instruction.
  • Linking the clusters to post-graduation job performance data would offer one way to check if the model captures industry-relevant abilities.
  • Patterns found in algorithm courses might appear in other CS topics, allowing a shared set of competency profiles across the curriculum.

Load-bearing premise

That converting programming and assignment data to xAPI and modeling it with Markov processes produces competency clusters accurate enough to guide real curriculum decisions without external validation against other measures.

What would settle it

An independent comparison of the derived student clusters against ratings of the same students' competencies by industry experts or standardized external tests.

Figures

Figures reproduced from arXiv: 2606.00200 by Ninghan Zheng, Shanshan Li, Weidong Liu, Yingqi Zhang.

Figure 1
Figure 1. Figure 1: CC2020 Conceptual Structure of the Competency [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Diagram of the xAPI Phrase Structure • Optional Elements: In addition to the base structure, xAPI includes optional elements such as context and results that provide additional information and documen￾tation of results for learning activities. • Learning Record Storage System (LRS): xAPI defines the LRS to store and manage learning behavior records in a unified format, thus enabling unified archiving, mana… view at source ↗
Figure 3
Figure 3. Figure 3: Student Behavior Sequential Markov Property [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: State Transition Matrices for Two Typical Student [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Student Competency Analysis Visualization Results [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of the Effect of Clustering Algorithm [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: shows the resultant data of the K-means algorithm clustering, where each row represents a subgroup (cluster), each row represents a competency feature, and the numbers and colors in the cells show the average feature values for that subgroup [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Graph Analyzing the Relationship Between Time of [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Correlation Analysis Between Proactiveness and Per [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Quantifying the Difficulty of Experiment Topic Sets [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
read the original abstract

This full research paper describes the assessment and presentation of student competencies in algorithm courses, grounded in the CC2020 competency model. With the growing emphasis on bridging the gap between academic training and industry demands, competency-based education, which integrates knowledge, skills, and dispositions, has become pivotal in computer science education. To bridge the gap, we need to develop a comprehensive framework to evaluate competencies (knowledge, skills, and dispositions) in computer science education. The research aims to analyze learning behavior patterns, design methods for competency assessment in algorithm courses, and evaluate the difficulty of course experiments to inform curriculum design. We collected programming experiment and written assignment data from 169 students, adapting it to the xAPI specification for unified analysis. In this work, Markov process modeling was employed to analyze behavioral sequences, revealing cognitive patterns during programming tasks. Multiple methods were applied to quantify competencies (knowledge, skills, dispositions) and identify distinct student clusters. Course difficulty was quantified using proactiveness metrics derived from submission timeliness. This work contributes a scalable framework for competency assessment in algorithm courses and offers actionable insights for personalized teaching and curriculum optimization. Practically, it enables instructors to tailor interventions based on student clusters and optimize task difficulty. Future work will integrate more students' performance to validate competency models and extend the framework to broader computer science curricula.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper claims to develop a comprehensive framework for evaluating competencies (knowledge, skills, and dispositions) in algorithm courses based on the CC2020 model. Using data from 169 students adapted to xAPI, it employs Markov process modeling for behavioral sequences, multiple quantification methods for competencies, clustering of students, and proactiveness metrics for experiment difficulty to provide insights for personalized teaching and curriculum optimization.

Significance. If the competency quantifications and clusters are accurate, the work offers a scalable, data-driven method for competency assessment in CS education, with potential for curriculum optimization and personalized interventions. The empirical approach with real student data and sequence analysis is a strength, though the lack of validation against external measures reduces the current significance.

major comments (1)
  1. [Abstract] Abstract: The claim of contributing 'a scalable framework for competency assessment in algorithm courses and offers actionable insights for personalized teaching and curriculum optimization' depends on the validity of the Markov modeling and clustering results in quantifying competencies. The abstract explicitly states that 'Future work will integrate more students' performance to validate competency models', indicating that no such validation is provided in the current manuscript. This is load-bearing as the practical utility for curriculum decisions rests on unverified correspondence between derived patterns and intended competencies.
minor comments (1)
  1. [Abstract] The abstract contains some repetitive phrasing regarding the gap between academic training and industry demands; streamlining this would improve readability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim of contributing 'a scalable framework for competency assessment in algorithm courses and offers actionable insights for personalized teaching and curriculum optimization' depends on the validity of the Markov modeling and clustering results in quantifying competencies. The abstract explicitly states that 'Future work will integrate more students' performance to validate competency models', indicating that no such validation is provided in the current manuscript. This is load-bearing as the practical utility for curriculum decisions rests on unverified correspondence between derived patterns and intended competencies.

    Authors: We agree that the manuscript provides no external validation of the derived competency quantifications or clusters against independent measures, as the abstract itself notes this as future work. The presented work is an initial framework that applies Markov modeling, multiple quantification methods, and clustering to behavioral data from 169 students; the resulting patterns are offered as preliminary insights rather than validated mappings to CC2020 competencies. We will revise the abstract to temper the contribution language, explicitly framing the work as an exploratory demonstration of the modeling approach with validation planned for future studies. revision: yes

Circularity Check

0 steps flagged

No circularity in empirical data analysis framework

full rationale

The paper presents an empirical study: data collection from 169 students, xAPI adaptation, Markov process modeling of sequences, and application of multiple methods to quantify and cluster competencies (knowledge, skills, dispositions) plus proactiveness-based difficulty metrics. No equations, self-citations, or derivations are described that reduce outputs to inputs by construction, nor any fitted parameters renamed as predictions. The central contribution is a data-driven framework whose validity rests on the (unvalidated) correspondence of derived clusters to competencies, but this is an external-validation issue rather than circularity. The derivation chain is self-contained as standard applied analysis without self-referential loops.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no specific free parameters, axioms, or invented entities are identifiable from the provided text. The work relies on standard external models (CC2020, xAPI, Markov processes) without detailing any ad-hoc elements.

pith-pipeline@v0.9.1-grok · 5769 in / 1225 out tokens · 29413 ms · 2026-06-28T19:41:51.786279+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references

  1. [1]

    Computing curricula 2005: The overview report,

    R. Shackelford, A. McGettrick, R. Sloan, H. Topi, G. Davies, R. Kamali, J. Cross, J. Impagliazzo, R. LeBlanc, and B. Lunt, “Computing curricula 2005: The overview report,” inProceedings of the 37th SIGCSE Tech- nical Symposium on Computer Science Education, ser. SIGCSE ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 456–457

    2005

  2. [2]

    Is educationrole of information systems in the cc2020 initia- tive,

    H. Topi, “Is educationrole of information systems in the cc2020 initia- tive,”ACM Inroads, vol. 8, no. 4, p. 43–44, oct 2017

    2017

  3. [3]

    C. T. Force,Computing Curricula 2020: Paradigms for Global Com- puting Education. New York, NY , USA: Association for Computing Machinery, 2020

    2020

  4. [4]

    The impact of the acm/ieee cc2020 competency model on the development of computer education in china,

    M. Zhang and J. Chen, “The impact of the acm/ieee cc2020 competency model on the development of computer education in china,”Computer Education, no. 04, pp. 3–8+14, 2023

    2023

  5. [5]

    A practical approach to assessing it professional skills,

    D. S. Bowers, M. Sabin, and R. K. Raj, “A practical approach to assessing it professional skills,” inProceedings of the 24th Annual Conference on Information Technology Education, ser. SIGITE ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 106–111

    2023

  6. [6]

    Using a professional skills framework to support the assessment of dispositions in it education,

    D. Bowers and M. Sabin, “Using a professional skills framework to support the assessment of dispositions in it education,” inProceedings of the 23rd Annual Conference on Information Technology Education, ser. SIGITE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 103–109

    2022

  7. [7]

    Cc2020 – visualization tool preview and review,

    A. Clear, S. Takada, and E. Cuadros Vargas, “Cc2020 – visualization tool preview and review,” inProceedings of the 51st ACM Technical Sym- posium on Computer Science Education, ser. SIGCSE ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 169–170

    2020

  8. [8]

    Cc2020 visualization tool,

    A. Clear, E. Cuadros-Vargas, and S. Takada, “Cc2020 visualization tool,” inProceedings of the 53rd ACM Technical Symposium on Computer Science Education V . 2, ser. SIGCSE 2022. New York, NY , USA: Association for Computing Machinery, 2022, p. 1063–1064

    2022

  9. [9]

    Research on the competency assessment method for students of the computer organization principles course,

    Y . Gao, “Research on the competency assessment method for students of the computer organization principles course,” Master’s thesis, Tsinghua University, 2023

    2023

  10. [10]

    Perspectives on dispositions in computing competencies,

    J. Impagliazzo, N. Kiesler, A. N. Kumar, B. Mackellar, R. K. Raj, and M. Sabin, “Perspectives on dispositions in computing competencies,” inProceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2, ser. ITiCSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 662–663

    2022

  11. [11]

    Construction of high- quality practical teaching courses for cybersecurity based on cc2020 competency model,

    Q. Lui, R. Wang, W. Zhao, J. Shi, and Y . Wang, “Construction of high- quality practical teaching courses for cybersecurity based on cc2020 competency model,”Computer Education, pp. 59–64, 2021

    2021

  12. [12]

    Optimization strategies for designing computer theory course assignments based on cc2020 competency model,

    Y . Liu, D. Yao, Y . Yang, X. Peng, and Y . Shi, “Optimization strategies for designing computer theory course assignments based on cc2020 competency model,”Computer Education, no. 07, pp. 199–204, 2022

    2022

  13. [13]

    The shift from static college textbooks to customizable content: A case study at zybooks,

    C. Gordon, R. L. Lysecky, and F. Vahid, “The shift from static college textbooks to customizable content: A case study at zybooks,”2021 IEEE Frontiers in Education Conference (FIE), pp. 1–7, 2021

    2021

  14. [14]

    Ultra-lightweight early prediction of at-risk students in cs1,

    C. Gordon, S. Zhao, and F. Vahid, “Ultra-lightweight early prediction of at-risk students in cs1,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 764–770

    2023

  15. [15]

    Experiences teaching coral before c++ in cs1,

    F. Vahid, K. Downey, L. Areizaga, and A. Pang, “Experiences teaching coral before c++ in cs1,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 340–345

    2023

  16. [16]

    Towards comprehensive metrics for programming cheat detection,

    F. Vahid, A. Pang, and B. Denzler, “Towards comprehensive metrics for programming cheat detection,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE

  17. [17]

    1361–1367

    New York, NY , USA: Association for Computing Machinery, 2024, p. 1361–1367

    2024

  18. [18]

    Impact of several low- effort cheating-reduction methods in a cs1 class,

    F. Vahid, K. Downey, A. Pang, and C. Gordon, “Impact of several low- effort cheating-reduction methods in a cs1 class,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 486–492

    2023

  19. [19]

    Experiences teaching a cs1 common course across 7 institutions,

    F. Vahid and A. Pang, “Experiences teaching a cs1 common course across 7 institutions,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1354–1360

    2024

  20. [20]

    Research on shared educational data model based on multi-source data fusion,

    F. Wu and S. Huang, “Research on shared educational data model based on multi-source data fusion,”Educational Technology Research, vol. 41, no. 05, pp. 59–65+103, 2020

    2020

  21. [21]

    Construction and empirical research on online learner profiles based on xapi,

    J. Xiao, H. Qiao, and X. Li, “Construction and empirical research on online learner profiles based on xapi,”China Educational Technology, pp. 123–129, 2019

    2019

  22. [22]

    https://codehs.com/,

    CodeHS, “https://codehs.com/,” 2022

    2022

  23. [23]

    Predicting mooc dropout over weeks using machine learning methods,

    M. Kloft, F. Stiehler, Z. Zheng, and N. Pinkwart, “Predicting mooc dropout over weeks using machine learning methods,” inProceedings of the EMNLP 2014 Workshop on Analysis of Large Scale Social Interaction in MOOCs, 01 2014, pp. 60–65

    2014

  24. [24]

    xapi——the new generation of learning technology specifications lead to new standards for smart education,

    Y . Tang, T. Zhao, and W. Wang, “xapi——the new generation of learning technology specifications lead to new standards for smart education,” Modern Educational Technology, vol. 25, no. 01, pp. 107–113, 2015

    2015

  25. [25]

    Obtaining educational big data: Acquisi- tion and sharing of learning experience data based on xapi specification,

    X. Gu, L. Zheng, and J. Jian, “Obtaining educational big data: Acquisi- tion and sharing of learning experience data based on xapi specification,” Modern Distance Education Research, no. 05, pp. 13–23, 2014

    2014

  26. [26]

    U. N. Bhat and G. K. Miller,Elements of applied stochastic processes. Wiley-Interscience Hobokenˆ eN. JNJ, 2002, vol. 3

    2002

  27. [27]

    G. A. Pavliotis,Stochastic Processes and Applications. Texts in applied mathematics, 2014, vol. 60

    2014

  28. [28]

    Chi-square tests for markov chain analysis,

    R. N. Hiscott, “Chi-square tests for markov chain analysis,”Journal of the International Association for Mathematical Geology, vol. 13, no. 1, pp. 69–80, 1981

    1981

  29. [29]

    Principal components analysis (pca),

    A. Ma ´ckiewicz and W. Ratajczak, “Principal components analysis (pca),” Computers & Geosciences, vol. 19, no. 3, pp. 303–342, 1993

    1993

  30. [30]

    Identifying different student clusters in functional programming assignments: From quick learners to struggling students,

    C. Geng, W. Xu, Y . Xu, B. Pientka, and X. Si, “Identifying different student clusters in functional programming assignments: From quick learners to struggling students,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE

  31. [31]

    New York, NY , USA: Association for Computing Machinery, 2023, p. 750–756

    2023

  32. [32]

    Some methods for classification and analysis of multi- variate observations,

    J. MacQueen, “Some methods for classification and analysis of multi- variate observations,” inProceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, vol. 5. University of California press, 1967, pp. 281–298

    1967

  33. [33]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.”Journal of machine learning research, vol. 9, no. 11, 2008

    2008