pith. sign in

arxiv: 1906.11421 · v1 · pith:HIRBBYTXnew · submitted 2019-06-27 · 💻 cs.FL · cs.PL

FSM Error Messages

Marco T. Moraz\'an (Seton Hall University) , Josephine A. Des Rosiers (Seton Hall University) This is my paper

Pith reviewed 2026-05-25 14:13 UTC · model grok-4.3

classification 💻 cs.FL cs.PL
keywords FSM libraryerror messagesautomata theorystate machinescomputer science educationdebuggingconstructive proofsstudent attitudes
0
0 comments X

The pith

A tailor-made error-messaging system for the FSM library leads to improved student attitudes toward automata theory and easier debugging of programs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper describes the development of custom error messages for the FSM library, which allows students to program state machines and grammars in the context of Automata Theory courses. Previously, error messages came from the host language and were hard to interpret without knowing the library's internals. A study with control group and survey shows that the new messages positively affect attitudes toward the subject, the library, and its errors, resulting in better ability to implement algorithms from constructive proofs.

Core claim

The results strongly suggest that the error-messaging system has had a positive impact on students' attitude towards automata theory, towards programming in FSM, and towards FSM error messages. The consequence has been a marked improvement on students' ability to implement algorithms developed as part of constructive proofs by making the debugging of FSM programs easier.

What carries the argument

The tailor-made error-messaging system for FSM, which provides messages designed specifically for users of the library rather than generic host-language errors.

If this is right

  • Students report more positive attitudes toward automata theory.
  • Students find programming in FSM more approachable.
  • Debugging FSM programs becomes easier, aiding implementation of algorithms from constructive proofs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar custom error systems might benefit other educational programming libraries used in theory courses.
  • Longer-term tracking could check whether improved attitudes lead to higher completion rates in automata theory classes.
  • Adoption at other institutions could test whether the observed effects hold beyond the original study setting.

Load-bearing premise

That the measured improvements in attitudes and debugging performance are attributable to the new error messages rather than to other unmeasured changes in instruction, student cohort, or study design.

What would settle it

A follow-up experiment that changes only the error messages while keeping all other instructional elements identical and finds no difference in attitudes or performance.

Figures

Figures reproduced from arXiv: 1906.11421 by Josephine A. Des Rosiers (Seton Hall University), Marco T. Moraz\'an (Seton Hall University).

Figure 1
Figure 1. Figure 1: FSM implementation of a deterministic finite-state machine to recognize a(a ∪ b)∗ a. informal discussions with students that state that if error messages were actually useful, then they would be willing to read them. 3 Overview of FSM The FSM library presents the user with a generic interface to construct and manipulate state machines and grammars. Constructors are divided into two categories: primitive co… view at source ↗
Figure 2
Figure 2. Figure 2: Buggy Turing machine that decides the language L = [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Student’s proposed context-free grammar for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A student’s transformer to remove unreachable states from a deterministic finite-state automa [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average Number of Unresolved Errors After Debugging. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overall Course Opinion. (R-ERRS). The second is for the study group using the library with FSM error messages (FSM-ERRS). The students were given one class period (i.e., 75 minutes) to debug the three functions. They were advised to spend at most 25 minutes on each program. For TMBB, the average number of unresolved errors after debugging is less than 1 for both groups. That is, most students in both group… view at source ↗
Figure 7
Figure 7. Figure 7: Level of Interest in Programming with FSM . note that students are specifically expressing an interest in programming with FSM. The lesson that needs to be derived is that even libraries for advanced courses benefit from a well-designed error-messaging system. Students in the second voyage of FSM at SHU were asked to evaluate their overall opinion of FSM error messages on a scale from 1 = A complete waste … view at source ↗
Figure 8
Figure 8. Figure 8: Overall Opinion on FSM Error Messages. Future work includes exploring how to improve the wording of individual messages. For example, should lots of detail be provided or should less detail be provided? Currently, FSM error messages provide lots of detail when machine rules contain type errors, but much less detail is provided for gram￾mar rules. More qualitative research is needed to determine if one stra… view at source ↗
read the original abstract

Computer Science students, in general, find Automata Theory difficult and mostly unrelated to their area of study. To mitigate these perceptions, FSM, a library to program state machines and grammars, was developed to bring programming to the Automata Theory classroom. The results of the library's maiden voyage at Seton Hall University had a positive impact on students, but the students found the library difficult to use due to the error messages generated. These messages were generated by the host language meaning that students needed to be familiar with the library's implementation to make sense of them. This article presents the design of and results obtained from using an error-messaging system tailor-made for FSM. The effectiveness of the library was measured by both a control group study and a survey. The results strongly suggest that the error-messaging system has had a positive impact on students' attitude towards automata theory, towards programming in FSM, and towards FSM error messages. The consequence has been a marked improvement on students' ability to implement algorithms developed as part of constructive proofs by making the debugging of FSM programs easier.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript describes the FSM library for programming state machines and grammars in Automata Theory courses and introduces a custom error-messaging system to replace opaque host-language errors. It reports that a control-group study and survey indicate the new system positively affected student attitudes toward automata theory, FSM programming, and error messages, while also improving debugging and implementation of algorithms from constructive proofs.

Significance. If the reported gains can be causally attributed to the error-messaging changes, the work would offer a concrete, deployable contribution to CS education tools that bridge theory and programming. The absence of methodological detail, however, prevents any assessment of whether the central claim holds.

major comments (2)
  1. [Abstract] Abstract: the claim that the error-messaging system produced positive impacts on attitudes and debugging performance rests on a 'control group study and a survey,' yet the abstract (and therefore the evidential basis presented) supplies no sample sizes, randomization procedure, description of the control condition, statistical tests, or effect sizes. This information is load-bearing for the causal attribution.
  2. [Methods / Results (control-group study)] Study design (control-group component): without explicit comparison to the prior library version (or other baseline), it is impossible to rule out confounds such as cohort differences, changes in instruction, or study-design artifacts. The weakest assumption identified in the stress-test note therefore remains unaddressed in the reported evidence.
minor comments (1)
  1. [Survey description] The manuscript would benefit from a dedicated subsection that tabulates the survey instrument items, response scales, and any pre/post or between-group comparisons performed.

Circularity Check

0 steps flagged

No circularity; empirical study reports survey and control-group outcomes without derivation chain

full rationale

The paper presents the design of FSM error messages and reports measured outcomes from a control-group study plus survey. No equations, parameters, predictions, or first-principles derivations appear; the central claims are direct empirical results rather than quantities derived from the same inputs. Patterns 1-6 do not apply. The paper is self-contained against external benchmarks (student attitude and performance data) and receives the default non-finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical education study; it introduces no mathematical free parameters, no domain axioms beyond standard assumptions of survey validity, and no new theoretical entities.

pith-pipeline@v0.9.0 · 5719 in / 990 out tokens · 30724 ms · 2026-05-25T14:13:33.626779+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

  1. [1]

    In: Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming, ICFP ’10, ACM, New York, NY , USA, pp

    Marcus Crestani & Michael Sperber (2010): Experience Report: Growing Programming Languages for Beginning Students. In: Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming, ICFP ’10, ACM, New York, NY , USA, pp. 229–234, doi:10.1145/1863543.1863576

    work page doi:10.1145/1863543.1863576 2010

  2. [2]

    Hsia, Elspeth Simpson, Daniel Smith & Robert Cartwright (2005): Taming Java for the Classroom

    James I. Hsia, Elspeth Simpson, Daniel Smith & Robert Cartwright (2005): Taming Java for the Classroom. In: Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education , SIGCSE ’05, ACM, New York, NY , USA, pp. 327–331, doi:10.1145/1047344.1047459

    work page doi:10.1145/1047344.1047459 2005

  3. [3]

    Lewis & Christos H

    Harry R. Lewis & Christos H. Papadimitriou (1997): Elements of the Theory of Computation , 2nd edition. Prentice Hall PTR, Upper Saddle River, NJ, USA, doi:10.1145/300307.1040360

    work page doi:10.1145/300307.1040360 1997

  4. [4]

    In: Proceedings of the 42Nd ACM Technical Sym- 16 FSM Error Messages posium on Computer Science Education , SIGCSE ’11, ACM, New York, NY , USA, pp

    Guillaume Marceau, Kathi Fisler & Shriram Krishnamurthi (2011): Measuring the Effectiveness of Er- ror Messages Designed for Novice Programmers . In: Proceedings of the 42Nd ACM Technical Sym- 16 FSM Error Messages posium on Computer Science Education , SIGCSE ’11, ACM, New York, NY , USA, pp. 499–504, doi:10.1145/1953163.1953308

    work page doi:10.1145/1953163.1953308 2011

  5. [5]

    In: Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , Onward! 2011, ACM, New York, NY , USA, pp

    Guillaume Marceau, Kathi Fisler & Shriram Krishnamurthi (2011): Mind Your Language: On Novices’ Interactions with Error Messages . In: Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , Onward! 2011, ACM, New York, NY , USA, pp. 3–18, doi:10.1145/2048237.2048241

    work page doi:10.1145/2048237.2048241 2011

  6. [6]

    Martin (2003): Introduction to Languages and the Theory of Computation , 3 edition

    John C. Martin (2003): Introduction to Languages and the Theory of Computation , 3 edition. McGraw-Hill, Inc., New York, NY , USA

    work page 2003

  7. [7]

    In: Proceedings of the 1996 International Conference on Software Engineering: Education and Practice (SE:EP ’96) , SEEP ’96, IEEE Computer Society, Washington, DC, USA, pp

    Linda McIver & Damian Conway (1996): Seven Deadly Sins of Introductory Programming Language Design. In: Proceedings of the 1996 International Conference on Software Engineering: Education and Practice (SE:EP ’96) , SEEP ’96, IEEE Computer Society, Washington, DC, USA, pp. 309–316, doi:10.1109/SEEP.1996.534015

    work page doi:10.1109/seep.1996.534015 1996

  8. [8]

    Moraz ´a & Rosario Antunez (2014): Functional Automata - F ormal Languages for Computer Science Students

    Marco T. Moraz ´a & Rosario Antunez (2014): Functional Automata - F ormal Languages for Computer Science Students . In James Caldwell, Philip K. F. H ¨olzenspies & Peter Achten, editors: Proceedings 3rd International Workshop on Trends in Functional Programming in Education , EPTCS 170, pp. 19–32, doi:10.4204/EPTCS.170.2

    work page doi:10.4204/eptcs.170.2 2014

  9. [9]

    Munson & Elizabeth A

    Jonathan P. Munson & Elizabeth A. Schilling (2016): Analyzing Novice Programmers’ Response to Compiler Error Messages. J. Comput. Sci. Coll. 31(3), pp. 53–61. Available at http://dl.acm.org/citation. cfm?id=2835377.2835386

    work page arXiv 2016

  10. [10]

    168–172, doi:10.1145/1352135.1352192

    Marie-H ´el`ene Nienaltowski, Michela Pedroni & Bertrand Meyer (2008): Compiler Error Messages: What Can Help Novices? In: Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, ACM, New York, NY , USA, pp. 168–172, doi:10.1145/1352135.1352192

    work page doi:10.1145/1352135.1352192 2008

  11. [11]

    https://www.heacademy.ac.uk/knowledge- hub/using-feedback-help-students-learn

    Phil Race (2005): Using Feedback to Help Students Learn . https://www.heacademy.ac.uk/knowledge- hub/using-feedback-help-students-learn

    work page 2005

  12. [12]

    Schiliep (2015): Usability of Error Messages for Introductory Students

    Paul A. Schiliep (2015): Usability of Error Messages for Introductory Students. Scholarly Horizons: Univer- sity of Minnesota, Morris Undergraduate Journal2(2), p. Article 5. Available athttp://digitalcommons. morris.umn.edu/horizons/vol2/iss2/5

    work page 2015

  13. [13]

    Cengage Learning

    Michael Sipser (2013): Introduction to the Theory of Computation , 3rd edition. Cengage Learning

    work page 2013

  14. [14]

    Javier Traver (2010): On Compiler Error Messages: What They Say and What They Mean

    V . Javier Traver (2010): On Compiler Error Messages: What They Say and What They Mean . Adv. in Hum.-Comp. Int. 2010, pp. 3:1–3:26, doi:10.1155/2010/602570

    work page doi:10.1155/2010/602570 2010