Pith.

Computers and Society · arXiv:2611.17220 v1 · submitted 12 Nov 2026 · 25 references

Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Geoffrey Challen, Craig Zilles

The paper says

“Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly.”

more of the abstract

In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Honest experience report on a two-agent LLM oral-exam system; the architecture and fictional-topic harness are genuinely new, but the grading validity gap is real and the paper's own audit shows it. This line is wrong

3 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 25 references say.

The paper says, at §5.1, §6

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.78):

“As such, this deployment should be interpreted as a demon- stration that the system provided a satisfactory experience for real students, rather than a validation that it is ready for high-stakes grading.”

Pith says: Objection 1 of 3 · major

The paper's central claim that conversational assessment is 'practical now' (§7) is undermined by the audit results reported in §5.1 and §6. The deployed evaluator achieved only 35% exact agreement with a blind re-grade, systematically awarded more E grades (20 vs. 2), and the key scaffolding rule (met vs. partially_met) was enforced least consistently.

more of objection 1

The authors acknowledge this but still assert practicality. The claim is supported for mechanics (cost, time, student engagement) but not yet for assessment validity. The paper should either qualify the claim explicitly to separate feasibility from grading reliability, or provide evidence that the proposed whole-transcript second read resolves the issue on the deployment transcripts. As written, the conclusion overstates what has been demonstrated.

The paper says, at §5.1, Table 2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.83):

“The persona harness tests archetypes, not the full continuum of real stu- dents.”

Pith says: Objection 2 of 3 · major

The validation harness (Table 2) uses only archetypal personas and does not include a persona that tests the met/partially_met boundary—the very distinction that later failed in deployment. The authors acknowledge this in §6 ('no persona covered' the leniency issue) and add a prompted-knower persona post-hoc.

more of objection 2

However, the pre-deployment validation therefore did not test the most load-bearing aspect of the rubric. The paper should state this limitation explicitly in the validation section (§4) and discuss how future validations can be designed to cover the scaffolding boundary before deployment.

The paper says, at §5.1, §6

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.63):

“As such, this deployment should be interpreted as a demon- stration that the system provided a satisfactory experience for real students, rather than a validation that it is ready for high-stakes grading.”

Pith says: Objection 3 of 3 · major

The audit establishes systematic disagreement between the deployed evaluator and post-hoc judges, but the paper does not resolve which grading is correct. The authors state that 'the audit establishes that the two vantage points disagree, not which is right' (§6).

more of objection 3

This is honest but leaves the central assessment-validity question open. The proposed solution (whole-transcript second read) is tested only on the prompted-knower persona, not on the 41 deployment transcripts. Without evidence that the second read would correct the leniency on real student data, the recommendation to adopt this architecture for even formative assessment rests on an unvalidated fix. The paper should either provide such evidence or explicitly defer the validity claim to future work.

7 minor objections

The paper says, at §3.1, Figure 1

Pith says: minor objection 1 of 7

The rubric example in Figure 1 uses 'met' vs. 'partially_met' based on scaffolding level, but the distinction between 'generic prompting' and 'specific questions' is subjective and may be inconsistently applied by the LLM. The paper could benefit from a more operational definition or examples of what constitutes generic vs. specific prompting.

The paper says, at §4.2

Pith says: minor objection 2 of 7

The fictional-topic insight is a strong methodological contribution. However, the paper does not discuss whether the fictional assessment's rubric structure is representative of real assessments. If the fictional topic is simpler or more constrained, the validation may not generalize. A brief note on this would strengthen the section.

show 5 more

The paper says, at §5.1

Pith says: minor objection 3 of 7

The agreement statistics (35% exact, 80% within one) are reported for the deployed evaluator vs. the Claude Fable 5 judge. The paper later shows that the model difference and grading approach are conflated (§5.1, [b147]). It would be clearer to present the decoupled results (production model post-hoc vs. deployed) first, then the cross-model comparison, to avoid initial misinterpretation.

The paper says, at §6

Pith says: minor objection 4 of 7

The phrase 'the post-hoc AI grades had similar variance to grading by experienced faculty members' (§6, [b162]) is not supported by a statistical comparison. The human-human agreement was 6/10 exact, AI-AI was 7/10 exact on the same ten transcripts. This is a small sample and the claim of 'similar variance' should be qualified or removed.

The paper says, at §5

Pith says: minor objection 5 of 7

The paper states that 'no student requested a re-grade' (§5) but also notes that grades were formative and carried no stakes. This observation is not informative about student satisfaction or perceived fairness. It should be interpreted cautiously.

The paper says, at §3.4

Pith says: minor objection 6 of 7

The implementation section mentions that the system was built 'substantially through collaboration with an LLM coding agent.' This is interesting but not essential. Consider moving to acknowledgments or a footnote.

The paper says, at §1, §7

Pith says: minor objection 7 of 7

The paper uses 'practical now' in the abstract and conclusion. Given the audit results, consider replacing with 'mechanically feasible now' or 'deployable at low cost, with grading reliability under active improvement' to avoid overstatement.

What would settle it

Re-grade all 40 deployed transcripts at the criterion level, not just at the overall grade: two independent human raters, blind to deployed grades, mark each student contribution as volunteered before detailed prompting or elicited by scaffolding, using the §3.1 rubric definitions; compare their criterion-level labels, and compare them to the evaluator's stored per-turn criterion states. If criterion-level agreement on the met/partially_met boundary is also near 35%, the scaffolding rule is not reliably applicable and the practical-now claim must be restricted to conversation mechanics; if criterion-level agreement is substantially higher, the deployed leniency is an aggregation or stopping-rule artifact and the second-read fix has a clear target.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 47 s · 13,767 tokens read, 6,257 written
  3. Editor wrote the verdict above 9 s · 14,683 tokens read, 527 written
  4. Referee major revision: 3 major objections, 7 minor 15 s · 14,160 tokens read, 1,636 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is an honest experience report, not a validation study, and it reads like one. The two-agent split (interviewer with no rubric access, hidden evaluator that steers) is a clean architectural contribution. The fictional-topic persona harness is a genuinely clever methodological fix for the problem that LLM-simulated students leak knowledge from pre-training. The paper is unusually transparent about its own limitations, including the fact that the deployed evaluator agreed with blind re-grades only 35% of the time and was systematically too lenient on the E/S boundary.

What the paper does well: it ships a working system, reports concrete cost figures ($0.27/attempt median), describes a regression-test pipeline for prompt changes, and documents an audit loop that feeds grading failures back into new personas. The prose is direct and the authors do not oversell the deployment evidence.

The soft spot is load-bearing. The central claim that conversational assessment is "practical now" rests on two legs: the mechanics work cheaply (true), and the grades measure what students can explain unprompted (unproven). The paper's own audit shows the deployed evaluator enforced the volunteered-vs-prompted distinction least consistently, with 35% exact agreement and 20 E's from the system versus 2 from the blind judge. The authors acknowledge this, but the conclusion still leans on the stronger claim. The evidence base is one small formative course (N=11, 41 attempts) with no grade-bearing stakes, and the code is not yet public.

Who this is for: builders of LLM-based assessment systems, especially those working on oral-exam analogues. The architecture and testing methodology are worth reading even if the validity evidence is thin. A serious referee would push on the grading validity gap and ask for a larger, higher-stakes deployment before accepting the strongest claims.

Recommendation: send to peer review. The paper is a genuine contribution to the engineering literature on conversational assessment, and the limitations are stated clearly enough that a good reviewer can help the authors scope their claims properly.

The strongest objection

The central claim in §7 is that conversational assessment is 'practical now.' That claim has two parts: the conversation machinery runs at low cost, and the resulting grade measures what the student can explain unprompted. The load-bearing rule is the met/partially_met boundary defined in §3.1: credit for a criterion requires knowledge volunteered before detailed prompting. §5.1 reports that the deployed evaluator enforced this rule least consistently: 35% exact agreement with a blind re-grade, 80% within one level, and 20 E's from the deployed evaluator versus 2 from the judge; both post-hoc judges agreed with the deployed grades only about a third of the time ([b144], [b149], [b150]). §6 reproduces the failure with a prompted-knower persona that should cap at S but received E, and notes that the evaluator leaked reference terminology into the interviewer's guidance. The paper acknowledges these facts and states that the audit establishes disagreement rather than which judge is right, yet the conclusion still says 'practical now.' The unresolved issue is not that AI and human grades differ; human-human exact agreement was 6/10 ([b153]). The unresolved issue is that the specific distinction that turns a conversation into an assessment—volunteered vs prompted knowledge—was the least reliable part of the measured system, and the proposed whole-transcript second read has not been validated on the deployment transcripts. Therefore the strongest claim is supported for mechanics and cost, but not for grade validity.

The reader's weakest assumption is the same one I would isolate: the E/S distinction on unprompted knowledge has to be reliably applied for the system to be an assessment rather than a scripted chat. The paper's own §5.1 and §6 provide direct evidence that this assumption failed in deployment, so I cannot report no significant objection. I do not think this changes the verdict: the paper is framed as an experience report, its limitations section already concedes the small scale and the absence of stakes, and the two-agent architecture plus the fictional-topic harness are genuine, separable contributions. The conditionality remains because the central claim extends from 'the system runs' to 'the system assesses,' and the audit data bear directly on that extension. A criterion-level human re-grade would distinguish between a fixable aggregation problem and a fundamental unreliability in applying the scaffolding rule; that distinction determines whether the next version can support grade-bearing use. No ad hominem is intended; the concern is about the evidence, and the paper itself supplies most of it.

The referee's report

If the central claim holds—that conversational assessment is practical now—the paper offers a concrete, replicable architecture for scaling oral exams, which have long been considered infeasible beyond small enrollments. The two-agent separation, fictional-topic validation harness, and audit-and-improve loop are methodological contributions that could generalize beyond this specific system. The paper is unusually transparent about its failures (35% agreement, systematic leniency) and does not overclaim readiness for high-stakes use. However, the significance is tempered by the fact that the most load-bearing component—reliable distinction between volunteered and prompted knowledge—remains unvalidated in deployment. The paper's value lies more in demonstrating a viable architecture and surfacing the calibration challenge than in proving assessment validity.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that the oral exam's core value — asking students to explain their reasoning and resolving ambiguity through follow-up questions — can be delivered at any enrollment by a two-agent LLM pipeline. A student-facing interviewer holds the conversation but never sees the rubric, while a hidden evaluator tracks progress, grades each turn, and decides what to ask next. The authors validated the design with eleven simulated students on a deliberately fictional topic, then deployed it in an eleven-student course, reporting twenty-minute conversations at a median cost of about $0.27 per attempt. Their own audit, however, found only 35% exact agreement between the deployed grades and blind re-grades, with the deployed evaluator too willing to grant the top grade because it failed to discount knowledge elicited by detailed follow-up questions. The paper's central claim is that conversational assessment is practical now, provided the volunteered-versus-prompted distinction at the heart of its rubric is continuously calibrated through persona regression tests and transcript audits.

What carries the argument. The central mechanism is a two-agent split combined with scaffolding-aware rubrics. The interviewer conducts the typed conversation with no access to the assignment specification, receiving only the evaluator's guidance about what to ask next; the evaluator updates structured per-turn state against the rubric and decides when to follow up and when to advance. The rubric's key distinction separates "met" from "partially met": credit goes only to knowledge the student volunteered before detailed prompting, which is also the rule that blocks answer extractors from earning credit. Validation runs through a fictional-topic harness — an invented procedure with no presence in any model's training data — so each persona knows exactly what its prompt says it knows, and the recommended operating loop is a persona sweep as a regression test before every prompt change, periodic audits of real transcripts, and new personas built from audit findings.

Core claim. The central claim is that an interactive, oral-exam-style assessment — where students explain and defend their reasoning and receive follow-up questions — can be run entirely by two cooperating LLM agents, at scale, for a negligible marginal cost. The load-bearing design choice is the separation between an interviewer that knows only the conversation and the evaluator's guidance, and a hidden evaluator that holds the rubric and reference material, so the interviewer cannot leak answers it does not have. Because the same evaluator examines every student, the format also avoids the multi-examiner inconsistency documented in the human grading literature. In validation, all five legitimate personas received their target grades, all six adversarial personas were scored Unsatisfactory, and 619 automated checks passed, using a fictional topic to keep persona knowledge free of pre-training contamination. In a real course, students completed 41 evaluated attempts with a median cost of $0.27; the deployment is presented as an experience report, not high-stakes validation. The after-the-fact audit showed the deployed evaluator graded too leniently, and the authors convert that finding into a new regression persona that reproduces the bug, recommending that the grade of record come from a whole-transcript second read rather than the per-turn state.

The premise it rests on. The system's grading validity rests on a single rule: an LLM evaluator reading a transcript can reliably tell knowledge the student volunteered from knowledge the interviewer had to elicit with specific questions — and the paper's own audit shows this rule was the least consistently enforced part of the deployed system, whose grades drifted systematically toward leniency.

If the paper is right (Pith's inferences, not the authors')

Conversational assessment is practical now: authoring is declarative, a conversation takes roughly twenty minutes regardless of enrollment, and the marginal cost per attempt is about a quarter.

Because the interviewer lacks the rubric and reference material, the adversarial personas — answer extractors, prompt injectors, confident fabricators, social engineers — all scored Unsatisfactory in validation, so the architecture blocks the most obvious cheating channel by construction.

The audit-and-improve loop gives an AI examiner a calibration mechanism that human examiners lack: persona sweeps catch prompt-change drift before it ships, and audit findings become new regression personas, such as the prompted knower that reproduced the documented leniency bug.

The proposed division of labor — per-turn grades for conversation management, a whole-transcript second read for the grade of record, and instructor review whenever the two disagree — is the paper's concrete remedy for the leniency it documented.

Because the deployed assessments never involved programming and the engine reads only a rubric from a text file, the architecture transfers to any discipline where explanations matter.

The fictional-topic validation method generalizes beyond this system: any test of an LLM interviewer, grader, or tutor could control persona knowledge by inventing a topic, and this becomes more valuable as pre-trained knowledge grows harder to audit.

The disagreement between the live evaluator and the retrospective judge suggests a design principle the paper does not state as a general law: in LLM-based assessment, the grade of record should come from the vantage point that did not steer the conversation, since the steering agent's per-turn judgments drifted toward leniency.

The $0.27 cost figure was measured on nineteen attempts in one small course; the claim that cost stays flat regardless of enrollment assumes current API pricing and concurrency limits hold, so the economics should be re-measured at several hundred concurrent attempts.

The blind judge's unprompted flag of a "marked register shift" hints that transcript-style anomalies could serve as a cheap statistical cheating detector in unproctored settings, an idea the paper mentions but does not develop.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-14

The authors' main claim, as the classifier picked it out 0.69

“In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.77 · human participants 0.88 · controlled comparison 0.05 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.10 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. background 1.00 0.07 0.03 asserted 1.00
But oral exams scale poorly. background 0.83 0.13 0.05 asserted 1.00
In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. contribution 0.99 0.91 0.23 asserted 1.00
We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. method 0.98 0.91 0.09 asserted 0.98
The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). method 0.97 0.94 0.03 asserted 1.00
A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. method 0.62 0.92 0.03 asserted 1.00
The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. method 0.72 0.95 0.03 asserted 1.00
Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed. result 0.45 0.97 0.11 asserted 0.98

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-14: 1 request, 6,380 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (2)
Figure 1 of the paper
Figure 1. Factual and opinion “portions” from the deployed
Figure 2 of the paper
Figure 2. The two-agent architecture. After each student mes

Figures are the authors', reproduced from the paper.

References (25)
  1. [1] Hunter M. Brimi. 2011. Reliability of Grading High School Work in English. Practical Assessment, Research & Evaluation16, 17 (2011). doi:10.7275/j531-fz38 on Pithdoi:10.7275/j531-fz38
  2. [2] Susan M. Brookhart, Thomas R. Guskey, Alex J. Bowers, James H. McMillan, Jeffrey K. Smith, Lisa F. Smith, Michael T. Stevens, and Megan E. Welsh. 2016. A Century of Grading Research: Meaning and Value in the Most Common Educational Measure.Review of Educational Research86, 4 (... on Pithdoi:10.3102/0034654316672069
  3. [3] Robyn M. Dawes, David Faust, and Paul E. Meehl. 1989. Clinical Versus Actuarial Judgment.Science243, 4899 (1989), 1668–1674. doi:10.1126/science.2648573 on Pithdoi:10.1126/science.2648573
  4. [4] Ethan Dickey, Ashok Saravanan, Alexander Siladie, Houyame Lkhider, Quion- driya Gee, and Andres Bejarano. 2026. GLOW: AI-Simulated Students Improve GTA Readiness. InProceedings of the 57th ACM Technical Symposium on Computer Science Education V. 2 (SIGCSE TS 2026). ACM, New Yo... on Pithdoi:10.1145/3770761.3777354
  5. [5] Walter Crosby Eells. 1930. Reliability of Repeated Grading of Essay Type Examina- tions.Journal of Educational Psychology21, 1 (1930), 48–52. doi:10.1037/h0071103 on Pithdoi:10.1037/h0071103
  6. [6] Hasmik Gharibyan. 2005. Assessing Students’ Knowledge: Oral Exams vs. Written Tests. InProceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2005). ACM, New York, NY, USA, 143–147. doi:10.1145/1067445.1067487 on Pithdoi:10.1145/1067445.1067487
  7. [7] William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz, and Chad Nelson on Pith
  8. [8] Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein. 2021.Noise: A Flaw in Human Judgment. Little, Brown Spark, New York, NY, USA on Pith
  9. [9] Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, Syed Ashraf, and Paul Denny. 2025. Evaluating Language Models for Generating and Judging Programming Feedback. InProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (SIGCS... on Pithdoi:10.1145/3641554.3701791
  10. [10] Marko Lubarda, Nathan Delson, Curt Schurgers, Maziar Ghazinejad, Saharnaz Baghdadchi, Alex Phan, Mia Minnes, Josephine Relaford-Doyle, Leah Klement, Carolyn Sandoval, and Huihui Qi. 2021. Oral Exams for Large-Enrollment Engineering Courses to Promote Academic Integrity and Stu... on Pithdoi:10.1109/fie49875.2021.9637124
  11. [11] I. C. McManus, M. Thompson, and J. Mollon. 2006. Assessment of Examiner Leniency and Stringency (’Hawk-Dove Effect’) in the MRCP(UK) Clinical Exami- nation (PACES) Using Multi-Facet Rasch Modelling.BMC Medical Education6, Article 42 (2006). doi:10.1186/1472-6920-6-42 on Pithdoi:10.1186/1472-6920-6-42
  12. [12] Marcus Messer, Neil C. C. Brown, Michael Kölling, and Miaojing Shi. 2025. How Consistent Are Humans When Grading Programming Assignments?ACM Transactions on Computing Education25, 4, Article 49 (2025). doi:10.1145/3759256 on Pithdoi:10.1145/3759256
  13. [13] Ed Novak and Peter Ohmann. 2023. Oral Exams in CS-Education: Pros and Cons in the Age of AI Assisted Programming.Journal of Computing Sciences in Colleges 39, 3 (2023), 32–33 on Pith
  14. [14] Ed Novak, Peter Ohmann, Scott Reckinger, and Shanon Reckinger. 2025. Oral Exams in Computer Science Education Amidst ChatGPT Dependency. InProceed- ings of the 56th ACM Technical Symposium on Computer Science Education V. 2 (SIGCSE TS 2025). ACM, New York, NY, USA, 1725. doi:1... on Pithdoi:10.1145/3641555.3705102
  15. [15] Peter Ohmann. 2019. An Assessment of Oral Exams in Introductory CS. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE 2019). ACM, New York, NY, USA, 613–619. doi:10.1145/3287324.3287489 on Pithdoi:10.1145/3287324.3287489
  16. [16] Peter Ohmann and Ed Novak. 2025. A Multi-Institutional Assessment of Oral Exams in Software Courses. InProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE TS 2025). ACM, New York, NY, USA, 882–888. doi:10.1145/3641554.3701848 on Pithdoi:10.1145/3641554.3701848
  17. [17] Scott J. Reckinger and Shanon Marie Reckinger. 2021. Oral Proficiency Exams in High-Enrollment Computer Science Courses. In2021 ASEE Virtual Annual Conference Content Access on Pith
  18. [18] Scott J. Reckinger and Shanon M. Reckinger. 2022. A Study of the Effects of Oral Proficiency Exams in Introductory Programming Courses on Underrepresented Groups. InProceedings of the 53rd ACM Technical Symposium on Computer Science Education, Volume 1 (SIGCSE 2022). ACM, New ... on Pith
  19. [19] Mihaela Sabin, Karen H. Jin, and Adrienne Smith. 2021. Oral Exams in Shift to Remote Learning. InProceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE 2021). ACM, New York, NY, USA, 666–672. doi:10.1145/3408877.3432511 on Pithdoi:10.1145/3408877.3432511
  20. [20] Iris Schleicher, Karsten Leitner, Jana Juenger, Andreas Moeltner, Miriam Rues- seler, Bernd Bender, Jasmina Sterz, Karl-Friedrich Schuettler, Sarah Koenig, and Joachim Gerhard Kreuder. 2017. Examiner Effect on the Objective Structured Clinical Exam – A Study at Five Medical Sc... on Pithdoi:10.1186/s12909-017-0908-1
show 5 more
  1. [21] Daniel Starch and Edward C. Elliott. 1912. Reliability of the Grading of High- School Work in English.The School Review20, 7 (1912), 442–457 on Pith
  2. [22] Daniel Starch and Edward C. Elliott. 1913. Reliability of Grading Work in History. The School Review21, 10 (1913), 676–681 on Pith
  3. [23] Daniel Starch and Edward C. Elliott. 1913. Reliability of Grading Work in Mathe- matics.The School Review21, 4 (1913), 254–259 on Pith
  4. [24] Chenyan Zhao, Max Fowler, Yael Gertner, Seth Poulsen, Matthew West, and Mariana Silva. 2026. AI-Supported Grading and Rubric Refinement for Free Response Questions. InProceedings of the 57th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE TS 2026). ACM, New ... on Pithdoi:10.1145/3770762.3772545
  5. [2000] Clinical Versus Mechanical Prediction: A Meta-Analysis.Psychological Assessment12, 1 (2000), 19–30. doi:10.1037/1040-3590.12.1.19 on Pithdoi:10.1037/1040-3590.12.1.19

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment