REVIEW 4 major objections 6 minor 27 references
The Goofus & Gallant Story Corpus for Practical Value Alignment
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper argues that a curated children's comic-strip corpus, where one character always acts well and the other always acts badly, gives machine learning models a grounded, low-noise source of social norms for value alignment.
desk verdict A small, honest multimodal dataset paper that is a plausible new resource, but the 'ideal dataset' claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Goofus & Gallant comic strip itself, a long-running children's feature whose two recurring characters perform opposite behaviors in the same situation. That structure turns into an automatic labeling machine: every action by the 'good' character becomes a normative example and every action by the 'bad' character becomes a non-normative example, with no manual moral judgment needed. On top of this, a 27-value taxonomy of social principles, derived by running a pretrained value model over the corpus texts, gives each example up to two principle labels, generated by an LLM and verified by human annotators. The baselines use transformer encoders for text and images, including a dual encoder that fuses both modalities, to demonstrate that the labels are learnable.
What would settle it
Take a sample of strips and ask a diverse group of adult readers to label each depicted action normative or non-normative; if a substantial share disagree with the Goofus/Gallant assignment, or if a model trained only on this corpus predicts human labels near chance on everyday scenarios never shown in the comic, the corpus's value-alignment claim is undercut.
Extended reading notes
Core claim
The central claim is that the corpus—not any single model—is the contribution: it provides a compact, deliberately designed collection of normative and non-normative behaviors, and this design makes it an appropriate training set for socially normative agents. The paper supports the claim in two ways. First, normativity labels are constructed from the comic's own structure, so no crowd worker has to decide what is good or bad. Second, the principle labels are anchored to a fixed 27-value taxonomy derived from a broader social-science value framework, with the final labels produced by repeated language-model prompting and checked by human reviewers. The baseline experiments are offered as evidence that the value information is actually learnable: models classify text-only normativity with high accuracy, and multimodal input helps; principle prediction works under a relaxed evaluation that accepts either of the two true principles.
Load-bearing premise
The whole dataset rests on the convention that every action by Gallant is normative and every action by Goofus is non-normative, with no independent check that the comic's intended lessons match the norms a deployed AI should actually follow.
Editorial extensions
If this is right
- A modest, purpose-built corpus can substitute for much larger crowdsourced or scraped datasets when the goal is learning everyday normative judgments.
- The 27-principle taxonomy makes value learning a bounded classification problem, which is more tractable for downstream agents than open-ended moral reasoning.
- Since both text and image versions are released, multimodal value-aligned systems can be trained and evaluated against a common benchmark.
- The corpus offers a testbed for whether models trained on children's stories generalize to adult everyday situations, and the paper invites further work on augmenting existing value-alignment systems with this data.
Reading between the lines
- The comic's paired-panel structure is a natural fit for contrastive learning—explicitly training a model to push normative and non-normative descriptions apart—but the paper's baselines treat examples independently; exploiting the pairs could improve sample efficiency.
- The normative ground truth is culturally specific: a children's magazine's etiquette from 1995 to 2017. A deployer should test transfer across countries, socioeconomic groups, and time before treating these labels as universal values.
- Because the 27-value taxonomy was selected by running a pretrained value model on the corpus, model bias could enter before human annotation; human verification checks agreement with predicted labels, not the adequacy of the taxonomy itself.
- A natural next step is to use the trained normativity classifier as a reward signal for reinforcement learning agents, turning the dataset from a benchmark into an actual alignment mechanism.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces the Goofus & Gallant Story Corpus, a multimodal dataset built from 1995-2017 Highlights magazine comic strips. The corpus has two components: the GnG Normative dataset (1,387 panel texts and 819 images) in which actions are labeled normative or non-normative based on whether the actor is Gallant or Goofus; and the GnG Principles dataset (819 examples) in which each action is assigned two social principles from a 27-value taxonomy derived from Kiesel et al. (2022), generated by GPT-4o with a limited human verification step. The paper reports baselines for two tasks: normativity classification (binary) and principles classification (multi-label), with best accuracies of 76.77% for the multimodal normativity model, 94.05% for text-only BERT normativity, and 74.84% relaxed accuracy for principles classification. The central claim is that, because the comics were deliberately designed to teach young children social principles, this corpus is an ideal dataset for training socially normative agents.
Significance. If the labels are valid, the corpus provides a compact, carefully designed, multimodal alternative to large crowdsourced moral datasets such as Moral Stories and Social Chemistry 101. The paper's release of the data and its transparent baselines are useful, and the multimodal pairing is an asset for future value alignment research. However, the significance is conditional: the strong performance of the baselines is unsurprising if the models are learning the comic's internal character convention, and the dataset's value depends on label validity and generalizability, both of which remain unverified. The paper also offers a useful contrast to sentiment analysis, showing that normativity is not reducible to sentiment, though this is a negative result.
major comments (4)
- [Section III.A and Section VII] The normative labels are assigned by character identity (Gallant = normative, Goofus = non-normative) and no human validation of these labels is reported. The paper itself identifies a counterexample in Section VII—'I think I broke your camera, Dad. I'm sorry'—where the breaking and the apology have different normative valences, so a single binary label is not well-defined for such composed actions. Because the proportion of ambiguous panels is not quantified, the 'ideal dataset' claim rests on an unverified premise. The authors should validate a sample of the normative labels with independent annotators, quantify ambiguous or mixed panels, and rerun the classifiers on a cleaned subset to measure the impact of label noise.
- [Section III.B.2.c and Section V.B] The GnG Principles labels are generated by GPT-4o and the human verification is constrained: reviewers select among the model's proposed principles or choose 'None,' with 93% agreement on at least one principle and 61% on both in a 100-sample check. This design cannot identify systematic omissions in the candidate set, and the principles classifier is trained and evaluated on these machine-generated labels. The relaxed evaluation rule (correct if the prediction matches either of the two machine-generated true principles) further inflates apparent accuracy. To support the principles dataset, the authors should create a fully human-annotated test set in which annotators choose from the full 27-value taxonomy without LLM suggestions, report inter-annotator agreement, and evaluate the model against that independent gold standard.
- [Table III and Section VI.A] The 94.05% text-only BERT accuracy on the GnG test set does not demonstrate that the model learned generalizable social norms; it may reflect dataset-specific stylistic differences between Goofus and Gallant panels even after name removal. The sentiment-analysis comparison only shows that normativity is not sentiment; it does not test generalization. The authors should evaluate the normativity classifier on external normative corpora (e.g., Moral Stories or ETHICS) or on a newly collected set of Goofus & Gallant panels with human-provided normative labels, and report error patterns on the ambiguous and mixed examples discussed in Section VII.
- [Abstract and Section IV] The claims that the corpus is 'an ideal dataset' and contains 'rich knowledge about socio-cultural norms and values' go beyond the evidence presented. The corpus reflects the values of a single children's magazine over 1995-2017, and the paper does not analyze the coverage, cultural specificity, or potential biases of those values. The authors should either moderate the claims to describe the corpus as a useful, deliberately designed resource, or add an analysis comparing the value distribution in the corpus with other normative datasets and discussing its limitations as a general value-alignment resource.
minor comments (6)
- [Section I] The word 'datasat' in the contributions summary should be 'dataset'.
- [Section III.B.2.b] The word 'assgined' should be 'assigned'.
- [Table I] Clarify whether the GnG Principles 'Text only' row refers to action descriptions, scene descriptions, or both; the row label is ambiguous.
- [Section III.B.1] Report how the majority-vote final label was chosen when no single principle received a majority, and discuss the relationship between the Fleiss kappa of 0.54 and the reliability of the final labels.
- [Section VI.B] The compliance information fed to the principles classifier is derived from the character-identity labels; state explicitly that this makes the principles task depend on the unvalidated normativity labels.
- [General] Add a data availability statement with the URL or access conditions for the dataset.
Circularity Check
Normativity labels are character identity by construction, and principle labels are GPT-generated and verified only against GPT's own candidates; baselines therefore measure learning of internal conventions rather than externally validated social values.
-
self definitional
[Section III.A (GnG Normative Dataset) and Section IV.A (Normativity Classification)]
"This setup provided us with an automatically labeled corpus where all actions done by Gallant are labeled as normative, and all actions done by Goofus are labeled as non-normative."
By the paper's own construction, 'normative' is defined as 'action performed by Gallant' and 'non-normative' as 'action performed by Goofus.' Section IV.A then states: 'We seek to show that knowledge of socially normative and non-normative behavior can be identified from naturally occurring stories.' The classification task is therefore, by definition, the task of identifying which of the two characters a panel depicts. After the names are replaced with pronouns, the residual visual and stylistic differences between Goofus and Gallant are sufficient to drive the reported accuracies (94% text-only BERT in Table III, >70% image and dual-encoder models in Table II) without the model learning any socially generalizable norm.
-
other
[Section III.B.2.b-c (Annotating Principles using LLMs; Human Verification) and Section V.B (Principles classification Models)]
"We provided three inputs in the prompt to the GPT model: the scene description, action description and compliance information. ... If the comic strip text features Gallant, the compliance information is set as followed, and for Goofus, it is violated. ... The reviewers were asked to select the principle they believed was violated or upheld by the action. If both principles seemed applicable, they were instructed to select both. If neither of the predicted principles was correct, they were to select 'None.'"
The 'ground truth' principle labels are the majority outputs of GPT-4o over five queries, produced with compliance information derived from the same Gallant/Goofus convention. The human 'verification' is not an independent annotation over the full principle taxonomy: reviewers choose only between the two candidate principles GPT-4o already proposed, and the reported agreement (93% for 'at least one principle') counts a label as correct whenever a human accepts either candidate. This cannot falsify GPT-4o's candidate set; if the correct principle was never proposed, the only possible response is 'None,' and that option is excluded from the 'at least one principle' agreement metric.
full rationale
The paper's central claim is that the Goofus & Gallant corpus is an ideal dataset for training socially normative agents because the comics were designed to teach social principles. That claim is supported by two internal evaluations, both of which reduce to the paper's own labeling conventions. First, the normativity label is defined by character identity: Gallant's actions are normative, Goofus's actions are non-normative. Training a classifier to predict this label is therefore equivalent to training it to identify character role, especially since names are removed but visual style, typical sentence constructions, and scenario patterns remain. The reported high accuracies, while real, do not establish that the model learned socially generalizable norms; they establish that the labeling rule is learnable. Second, the principles dataset is generated by GPT-4o with compliance information tied to the same character convention, and the human review only selects among GPT-4o's proposed candidates. The 91-93% agreement figures are self-confirming because the human's choice set is the model's own output. Baselines trained on these labels thus measure consistency with a model-generated annotation process, not agreement with an independent value standard. The paper is transparent about several limitations—small size, coarse binary normativity, and moderate Fleiss kappa on the crowdsourced principles—but it does not provide an external benchmark or independent human labeling of the full taxonomy against which the GPT-4o labels are checked. These are not cases of self-citation or imported uniqueness theorems; they are cases where the 'prediction' is defined in terms of the construction that produced the label. The dataset may still be useful as a design-intent resource, but the circularity in the internal validation prevents the strong claim that the corpus contains 'rich knowledge about socio-cultural norms and values' from being demonstrated by the reported experiments.
Assumptions & free parameters
free parameters (1)
- Social principle label set size =
27 values
assumptions (4)
- domain assumption Automated normativity labels: all Goofus actions are non-normative, all Gallant actions are normative.
- domain assumption GPT-4o principle predictions, verified by constrained human review, are treated as ground truth.
- domain assumption The Kiesel et al. value taxonomy is an appropriate and complete basis for social principles.
- domain assumption Children's comics designed to teach social norms are a better training source than larger crowdsourced corpora.
Cite this review
Pith. "Pith review of The Goofus & Gallant Story Corpus for Practical Value Alignment." pith.science (2026). https://pith.science/paper/H36VN5FV
@misc{pith2026250109707,
author = {Pith},
title = {Pith review of: The Goofus & Gallant Story Corpus for Practical Value Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/H36VN5FV}},
note = {Machine review of arXiv:2501.09707}
}
read the original abstract
Values or principles are key elements of human society that influence people to behave and function according to an accepted standard set of social rules to maintain social order. As AI systems are becoming ubiquitous in human society, it is a major concern that they could violate these norms or values and potentially cause harm. Thus, to prevent intentional or unintentional harm, AI systems are expected to take actions that align with these principles. Training systems to exhibit this type of behavior is difficult and often requires a specialized dataset. This work presents a multi-modal dataset illustrating normative and non-normative behavior in real-life situations described through natural language and artistic images. This training set contains curated sets of images that are designed to teach young children about social principles. We argue that this is an ideal dataset to use for training socially normative agents given this fact.
Figures
Reference graph
Works this paper leans on
-
[1]
Cooper- ative inverse reinforcement learning,
D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooper- ative inverse reinforcement learning,” Advances in neural information processing systems, vol. 29, 2016
work page 2016
-
[2]
Aligning superintelligence with human interests: A technical research agenda,
N. Soares and B. Fallenstein, “Aligning superintelligence with human interests: A technical research agenda,” Machine Intelligence Research Institute technical report , vol. 8, 2014
work page 2014
-
[3]
Research priorities for robust and beneficial artificial intelligence,
S. Russell, D. Dewey, and M. Tegmark, “Research priorities for robust and beneficial artificial intelligence,” Ai Magazine , vol. 36, no. 4, pp. 105–114, 2015
work page 2015
-
[4]
Value alignment or mis- alignment - what will keep systems accountable?
T. Arnold, D. Kasenberg, and M. Scheutz, “Value alignment or mis- alignment - what will keep systems accountable?” in AAAI Workshop: AI, Ethics, and Society , 2017
work page 2017
-
[5]
S. J. Russell, Human Compatible: Artificial Intelligence and the Problem of Control. Viking (October 8, 2019), 2019
work page 2019
-
[6]
The nature, importance, and difficulty of machine ethics,
J. H. Moor, “The nature, importance, and difficulty of machine ethics,” IEEE intelligent systems , vol. 21, no. 4, pp. 18–21, 2006
work page 2006
-
[7]
Efficient supervision for robot learning via imitation, simulation, and adaptation,
M. Wulfmeier, “Efficient supervision for robot learning via imitation, simulation, and adaptation,” KI - K ¨unstliche Intelligenz, pp. 1–5, 2019
work page 2019
-
[8]
Third-person imitation learning,
B. C. Stadie, P. Abbeel, and I. Sutskever, “Third-person imitation learning,” in 5th International Conference on Learning Representations, ICLR 2017, 2017
work page 2017
Show all 27 references
-
[9]
Moral stories: Situated reasoning about norms, intents, actions, and their consequences,
D. Emelin, R. Le Bras, J. D. Hwang, M. Forbes, and Y . Choi, “Moral stories: Situated reasoning about norms, intents, actions, and their consequences,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Online and Punta Cana, Dominican ...
2021
-
[10]
Aligning AI with shared human values,
D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt, “Aligning AI with shared human values,” CoRR, vol. abs/2008.02275, 2020
2008 arXiv
-
[11]
Can machines learn morality? the delphi experiment,
L. Jiang, J. D. Hwang, C. Bhagavatula, R. L. Bras, J. Liang, J. Dodge, K. Sakaguchi, M. Forbes, J. Borchardt, S. Gabriel et al. , “Can machines learn morality? the delphi experiment,” arXiv preprint arXiv:2110.07574, 2021
2021 arXiv
-
[12]
Identifying the human values behind arguments,
J. Kiesel, M. Alshomary, N. Handke, X. Cai, H. Wachsmuth, and B. Stein, “Identifying the human values behind arguments,” in Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Dublin, Ireland: Association for Comp...
2022
-
[13]
The value learning problem,
N. Soares, “The value learning problem,” Machine Intelligence Research Institute, Berkley, 2015
2015
-
[14]
A survey on hate speech detection using natural language processing,
A. Schmidt and M. Wiegand, “A survey on hate speech detection using natural language processing,” in Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 1–10
2017
-
[15]
Detection and fine-grained classification of cyberbullying events,
C. Van Hee, E. Lefever, B. Verhoeven, J. Mennes, B. Desmet, G. De Pauw, W. Daelemans, and V . Hoste, “Detection and fine-grained classification of cyberbullying events,” in Proceedings of the Interna- tional Conference Recent Advances in Natural Language Processing . Hissar, B...
2015
-
[16]
Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter,
S. V olkova, K. J. Shaffer, J. Y . Jang, and N. O. Hodas, “Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter,” 7 2017. [Online]. Available: https://www.osti.gov/biblio/1373869
2017
-
[17]
Man is to computer programmer as woman is to homemaker? debiasing word embeddings,
T. Bolukbasi, K.-W. Chang, J. Zou, V . Saligrama, and A. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” in Proceedings of the 30th International Conference on Neural Information Processing Systems , ser. NIPS’16. Red Hook, NY , USA:...
2016
-
[18]
Social bias frames: Reasoning about social and power implications of language,
M. Sap, S. Gabriel, L. Qin, D. Jurafsky, N. A. Smith, and Y . Choi, “Social bias frames: Reasoning about social and power implications of language,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Online: Association for Computation...
2020
-
[19]
Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes,
N. Lourie, R. L. Bras, and Y . Choi, “Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes,” CoRR, vol. abs/2008.09094, 2020. [Online]. Available: https://arxiv.org/abs/ 2008.09094
2008 arXiv
-
[20]
Social chemistry 101: Learning to reason about social and moral norms,
M. Forbes, J. D. Hwang, V . Shwartz, M. Sap, and Y . Choi, “Social chemistry 101: Learning to reason about social and moral norms,” CoRR, vol. abs/2011.00620, 2020. [Online]. Available: https://arxiv.org/abs/2011.00620
2011 arXiv
-
[21]
Delphi: Towards machine ethics and norms,
L. Jiang, J. D. Hwang, C. Bhagavatula, R. L. Bras, M. Forbes, J. Bor- chardt, J. Liang, O. Etzioni, M. Sap, and Y . Choi, “Delphi: Towards machine ethics and norms,” 2021
2021
-
[22]
Measuring nominal scale agreement among many raters
J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological bulletin, vol. 76, no. 5, p. 378, 1971
1971
-
[23]
Gpt-4: Technical report,
OpenAI, “Gpt-4: Technical report,” 2024. [Online]. Available: https: //openai.com/index/hello-gpt-4o/
2024
-
[24]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020. [Onlin...
2010 arXiv
-
[25]
BERT: Pre- training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...
2019
-
[26]
Distilbert base uncased finetuned sst-2,
“Distilbert base uncased finetuned sst-2,” https://huggingface.co/ distilbert/distilbert-base-uncased-finetuned-sst-2-english
-
[27]
Recursive deep models for semantic compositionality over a sentiment treebank,
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing . Seattle, Washington, USA:...
2013
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.