ComicScene154: A Scene Dataset for Comic Analysis

Ivan P. Yamshchikov; Pascal Mei{\ss}ner; Sandro Paval

REVIEW 3 major objections 6 minor 28 references

ComicScene154: A Scene Dataset for Comic Analysis

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ComicScene154 introduces manually labeled scene boundaries for 154 pages of public-domain comics, making narrative segmentation in comics a measurable task.

desk verdict A genuinely new but small comic scene dataset with honest baselines; the main risk is that the gold-standard annotations are under-validated. read the letter →

arxiv 2508.16190 v1 pith:36JMTXME submitted 2025-08-22 cs.CL

Sandro Paval , Ivan P. Yamshchikov , Pascal Mei{ss}ner This is my paper · ORCID

classification cs.CL

keywords comicscenesegmentationnarrativearcsmultimodalunderstandingdatasetbenchmarkpublic-domaincomicsinter-annotatoragreementpkmetricboundarydetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ComicScene154, a manually annotated dataset of 154 pages from four public-domain comic magazines in which each panel is marked as starting a new scene or not. Scenes are defined as plot-based semantic units where a cast pursues an overarching task, borrowing a definition from movie scene segmentation and treating scenes as narrative arcs. The authors argue this fills a gap: existing comic datasets annotate pages, objects, or characters, not narrative structure, and page-level divisions miss story boundaries. To show the dataset is usable, they build a two-stage baseline—a multimodal model that proposes scene boundaries and a reasoning language model that refines them—and evaluate with the pk segmentation metric, which measures how often two segmentations disagree on sliding windows. Their results show human annotators agree moderately (average pk 0.17 versus roughly 0.4 for random boundaries), while the baseline remains near random, which they read as evidence that scene segmentation is a hard, still-subjective task that the dataset can now make measurable.

What carries the argument

The panel-level scene-start tag: each panel in the 154 pages is numbered in reading order and labeled with a Boolean indicating whether it begins a new scene, or narrative arc. This single mechanism converts the abstract notion of 'scene' into a countable segmentation task, measurable by the pk sliding-window agreement metric adapted from text segmentation, and comparable across human annotators and model outputs.

What would settle it

Recruit independent annotators to label all 154 pages and compute pairwise pk among them, then compare model-to-label pk against this human-human agreement. If human-human pk on the full dataset approaches the random baseline of 0.4 rather than the reported 0.17, the scene boundaries are not a stable ground truth; if model scores match human-human agreement, the benchmark is already at the ceiling set by annotation subjectivity.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that ComicScene154 provides a comic dataset labeled at the level of narrative scenes rather than pages or panels, and that this level is the right granularity for multimodal narrative understanding. The discovery is the annotation resource itself: 34 full stories from four Golden Age comics, totaling 154 pages, with each panel numbered in reading order and carrying a Boolean scene-start label. The paper does not claim its baseline solves scene segmentation; it claims the dataset makes the task well-defined and benchmarkable, and documents that both humans and a state-of-the-art multimodal-plus-reasoning pipeline settle only slightly above random

Load-bearing premise

The authors' own manual scene annotations, used as ground truth for all evaluations, are reliable enough to serve as a benchmark even though independent annotators agree with them only mildly (average pk of 0.17 on one-third of the dataset).

Editorial extensions

If this is right

Scene segmentation in comics can be evaluated quantitatively with the pk metric, giving future work an objective benchmark rather than page-level heuristics.
Annotating full stories rather than random pages is necessary for scene construction, because narrative arcs cross page boundaries.
Human annotators agree moderately (average pk 0.17 against roughly 0.4 for random boundaries), but the lack of an intersubjective scene definition remains the core obstacle.
A multimodal model plus reasoning-based refinement performs only slightly better than random scene boundaries (average pk 0.39-0.46), despite high self-consistency across iterations.
The dataset provides a foundation for story summarization, character identification, and entity tracking at narrative scale, and for applying comic-style compression to movie scene segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because human annotators disagree substantially, the dataset's reliability could be improved by reporting multiple annotation views or a consensus labeling rather than a single ground truth; the authors leave this implicit.
The comic-as-video-compression analogy suggests a direct transfer test: sample movie frames into comic-like panel sequences and see whether scene boundaries learned on comics predict movie scenes.
The measured inter-annotator agreement of pk 0.17 sets an upper ceiling on how closely any model can match the official labels; future evaluations should compare model scores against human-human agreement, not just against random segmentation.
The Golden Age source material limits the benchmark to older storytelling and art styles, so extending it to modern comics would test how well the scene construct generalizes; the authors acknowledge this limitation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is the first labeled scene-segmentation dataset for comics, and that is the real contribution. The paper is honest about its limits and reports baseline performance that is barely above random. Second thing: the annotations that are the whole point of the dataset are not as solid as they need to be. pk=0.17 is reported without any chance-adjusted measure, so I can't tell whether that's actually good agreement or just above noise. That's the soft spot to worry about.

What's new and done well: the dataset fills a gap. Table 1 shows no prior comic dataset has scene-level annotations. They use public-domain sources, extract panels in reading order, and provide binary scene-start labels. The construction is transparent. They measure inter-annotator agreement with pk, and the average 0.17 is clearly better than the random baseline of 0.42 they report, though that comparison isn't apples-to-apples. Baselines using a multimodal model plus LLM refinement are reported as near-random and they don't oversell them. No sign of circular reasoning; the only mildly circular bit is computing k from the same annotations used for evaluation, but that's a minor issue.

Soft spots: the load-bearing assumption is annotation reliability. Only one-third of the data has multi-annotator labels. One excerpt hit 0.37, which is close to random. There's no Cohen's kappa or Krippendorff's alpha, and no boundary-level precision/recall. The pk=0.17 number is not interpretable without knowing the chance level for this boundary distribution and window size. Also, the annotation guideline in Appendix 7.2 doesn't state the formal scene definition from Section 2.1; it asks annotators to mark transitions intuitively. That raises a question about whether the labels actually reflect the 'narrative arc' construct the paper claims. The small size and Golden Age skew are acknowledged limitations, not hidden. The refined baseline's best-iteration results are cherry-picking, but they report averages too.

Who should read this: anyone working on comic analysis, multimodal narrative, or segmentation evaluation. The dataset is a useful seed even with small scale. It deserves a serious referee; the right fix is for the authors to add chance-adjusted agreement metrics, report per-boundary agreement, and align the annotation guideline with the formal definition. I'd take it in with revisions, not reject it.

Referee Report

3 major / 6 minor

Summary. The paper presents ComicScene154, a manually annotated dataset for scene-level narrative segmentation in comics. It comprises 154 pages from four public-domain comics (1942–1962), 34 stories, and roughly 954 panels, with panel-level binary labels marking scene starts. Inter-annotator reliability is measured on one-third of the data using the pk metric (average tester-vs-author pk = 0.17; tester-tester pk = 0.21). The authors also propose a two-stage baseline using Gemini 2.0 Flash Thinking to predict scene boundaries and then refine them; the baseline performs close to random (average pk 0.42 vs random 0.46), with refinement improving the best iterations (average 0.34) but not the average. The paper argues the dataset is a valuable resource for multimodal narrative understanding despite subjectivity and the Golden-Age skew.

Significance. If the gold annotations are accepted as reliable, ComicScene154 fills a clear gap: no existing comic dataset provides narrative scene segmentation with panel-level boundaries, and the task connects to video scene segmentation and semantic text segmentation. Strengths include public-domain reproducible sourcing, an explicit inter-annotator reliability effort, honest reporting of near-random baseline results, and no parameter tuning to inflate scores. The dataset and code are released. The main risk is that the reliability evidence is too thin to support the benchmark claim, and the construct validity of the annotation guidelines is questionable. These concerns are addressable and do not require collecting a fundamentally different dataset.

major comments (3)

[Section 3.3, Tables 3–4] The central claim that ComicScene154 is a reliable benchmark rests on the inter-annotator agreement, but only pk is reported (avg 0.17 tester-vs-author; 0.21 tester-tester). pk is not chance-adjusted; without the boundary density of the gold annotations and a random-segmentation baseline computed against the same gold, pk=0.17 cannot be interpreted. The random baseline in Table 4 is for model outputs vs human gold, not for human annotators vs human gold, so the Discussion's statement that human agreement is a 'notable improvement compared to randomly defined scenes' (§5) is not directly supported. Please report chance-level pk, a chance-corrected coefficient, boundary precision/recall, and a human-vs-human random baseline.
[Appendix 7.2 / Section 3.3] The annotation guideline given to testers does not contain the formal scene definition of §2.1 (a plot-based semantic unit pursuing an overarching task with a certain cast, analogous to Cohn's narrative arcs). It only asks participants to mark perceived transitions and says deviations are acceptable. Consequently, the gold-standard labels may operationalize intuitive page/panel segmentation rather than the intended narrative-arc construct. At minimum, include the formal definition in the guideline and describe how testers were briefed; ideally add a validation of the construct, e.g., whether scene boundaries correlate with changes in characters or goals.
[Section 4, Table 4] The 'random' baseline is under-specified. 'Scenes we defined randomly' does not state how many boundaries were sampled, whether the number of scenes matched the human annotations, or what distribution was used. Since the model's advantage over random is only 0.03–0.04 pk, the conclusion that the model is 'marginally better than random' depends entirely on this baseline. Provide the exact random-generation procedure and, ideally, multiple random draws with confidence intervals.

minor comments (6)

[Section 3.3] The formula '0.17 = 0.15 + 0.19/2' should read '(0.15 + 0.19)/2'; as written the arithmetic is incorrect.
[Table 3] Define what the 'In-between' column denotes (author–tester vs tester–tester) in the caption; currently the reader must infer it from §3.3.
[Section 3.3] The phrase 'three different groups of two annotators' is confusing in light of Table 3's '2 of 6 tester'; clarify how many annotators participated and how excerpts were assigned.
[Section 3.3] The window size k=3 is derived from the same annotations used for evaluation. Report the sensitivity of the reported pk values to k (e.g., k=2,4).
[Table 2 / References] Typos: 'Details on of' in the Table 2 caption; 'comptutational' in the Introduction; 'V olume' in the Kamath et al. reference.
[General] The dataset link is a GitHub repository; for archival stability, provide a DOI or a permanent repository entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset, annotations, and baseline evaluation are self-contained and do not reduce to their own inputs.

full rationale

ComicScene154 is a dataset contribution rather than a derivation chain. The ground-truth scene annotations are human-produced labels; the baseline is a frozen external multimodal model (Gemini) evaluated against those labels without any parameter fitting or test-set tuning, so the reported pk scores are not fitted values renamed as predictions. The only potentially questionable step is computing the pk window size k=3 from the same annotations used for evaluation (Section 3.3), but this is an operational choice for the metric, not a construction of the result: the human agreement and model scores are not forced by this choice, and the paper's central claim (that the dataset is a useful benchmark) does not reduce to it. The paper contains no load-bearing self-citations, and its limitations section honestly acknowledges the subjectivity and modest benchmark performance. Thus there is no circular step meeting the evidentiary standard.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted model parameters. Its contributions are data and an evaluation framework, both resting on imported definitions and metrics.

free parameters (1)

pk window size k = 3
Computed as half of the average scene length using the authors' and testers' annotations (Section 3.3). This is a data-dependent choice for the pk metric, but it is standard.

assumptions (3)

domain assumption A scene is defined as a plot-based semantic unit pursuing an overarching task with a certain cast, generally with temporal and spatial coherence (Rao et al., 2020), and is analogous to narrative arcs (Cohn, 2013).
The entire annotation task rests on this borrowed definition, which may not transfer perfectly to comics.
domain assumption The pk metric from text segmentation is a valid measure of segmentation agreement for comic panel sequences.
The paper applies pk without validating its behavior on panel-level segmentation.
domain assumption Public-domain Golden Age comics (1940-1962) are a sufficient source for a representative scene segmentation dataset.
The dataset is restricted to four magazines from this era, limiting generality (acknowledged in Limitations).

how reviews work

0 comments

Cite this review

Pith. "Pith review of ComicScene154: A Scene Dataset for Comic Analysis." pith.science (2026). https://pith.science/paper/36JMTXME

@misc{pith2026250816190,
  author       = {Pith},
  title        = {Pith review of: ComicScene154: A Scene Dataset for Comic Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36JMTXME}},
  note         = {Machine review of arXiv:2508.16190}
}

read the original abstract

Comics offer a compelling yet under-explored domain for computational narrative analysis, combining text and imagery in ways distinct from purely textual or audiovisual media. We introduce ComicScene154, a manually annotated dataset of scene-level narrative arcs derived from public-domain comic books spanning diverse genres. By conceptualizing comics as an abstraction for narrative-driven, multimodal data, we highlight their potential to inform broader research on multi-modal storytelling. To demonstrate the utility of ComicScene154, we present a baseline scene segmentation pipeline, providing an initial benchmark that future studies can build upon. Our results indicate that ComicScene154 constitutes a valuable resource for advancing computational methods in multimodal narrative understanding and expanding the scope of comic analysis within the Natural Language Processing community.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

[1]

Khaled Bayoudh, Raja Knani, Fay c al Hamdaoui, and Abdellatif Mtibaa. 2022. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 38(8):2939--2970

work page 2022
[2]

Neil Cohn. 2010. The limits of time and transitions: Challenges to theories of sequential image comprehension. Studies in Comics, 1(1):127--147

work page 2010
[3]

Neil Cohn. 2011. A different kind of cultural frame: An analysis of panels in american comics and japanese manga. Image & Narrative, 12(1):120--134

work page 2011
[4]

Neil Cohn. 2013. The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. Bloomsbury Academic, New York

work page 2013
[5]

Alexander Dunst, Rita Hartel, and Jochen Laubrock. 2017. The graphic narrative corpus (gnc): design, annotation, and analysis for the digital humanities. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 3, pages 15--20. IEEE

work page 2017
[6]

Arpita Dutta, Samit Biswas, and Amit Kumar Das. 2022. Bcbid: first bangla comic dataset and its applications. International Journal on Document Analysis and Recognition (IJDAR), 25(4):265--279

work page 2022
[7]

Azuma Fujimoto, Toru Ogawa, Kazuyoshi Yamamoto, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2016. Manga109 dataset and creation of metadata. In Proceedings of the 1st international workshop on comics analysis, processing and understanding, pages 1--5

work page 2016
[8]

Goran Glava s , Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation using semantic relatedness graphs. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 125--130. Association for Computational Linguistics

work page 2016

Show all 28 references

[9]

Cl \'e ment Gu \'e rin, Christophe Rigaud, Antoine Mercier, Farid Ammar-Boudjelal, Karell Bertet, Alain Bouju, Jean-Christophe Burie, Georges Louis, Jean-Marc Ogier, and Arnaud Revel. 2013. ebdtheque: a representative database of comics. In 2013 12th International Conference o...

2013
[10]

Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association...

2016
[11]

Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daume, and Larry S Davis. 2017. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In Proceedings of the IEEE Conference on Computer Vision and Patt...

2017
[12]

Apurva Kamath, Rishiraj Das, et al. 2019. A survey on semantic parsing. ACM Computing Surveys

2019
[13]

Najoung Kim and Sebastian Schuster. 2023. Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835--3855

2023
[14]

Yingxuan Li, Ryota Hinami, Kiyoharu Aizawa, and Yusuke Matsui. 2024. Zero-shot character identification and speaker prediction in comics via iterative multimodal fusion. arXiv preprint arXiv:2404.13993

2024 arXiv
[15]

Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2018. Digital comics image indexing based on deep learning. Journal of Imaging, 4(7):89

2018
[16]

Toru Ogawa, Atsushi Otsubo, Rei Narita, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018. Object detection for comics using manga109 annotations. arXiv preprint arXiv:1803.08670

2018 arXiv
[17]

Henry John Pratt. 2009. Narrative in comics. The Journal of Aesthetics and Art Criticism, 67(1):107--117

2009
[18]

Ben Proven-Bessel, Zilong Zhao, and Lydia Chen. 2021. Comicgan: Text-to-comic generative adversarial network. arXiv preprint arXiv:2109.09120

2021 arXiv
[19]

Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. 2020. A local-to-global approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

2020
[20]

Christophe Rigaud, Jean-Christophe Burie, and Samuel Petit. 2024. Toward accessible comics for blind and low vision readers. In International Conference on Document Analysis and Recognition, pages 198--215. Springer

2024
[21]

Ragav Sachdeva and Andrew Zisserman. 2024. The manga whisperer: Automatically generating transcriptions for comics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12967--12976

2024
[22]

Emanuele Vivoli, Marco Bertini, and Dimosthenis Karatzas. 2024. Comix: A comprehensive benchmark for multi-task comic understanding. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track

2024
[23]

Meng Wang, Richang Hong, Xiao-Tong Yuan, Shuicheng Yan, and Tat-Seng Chua. 2012. Movie2comics: Towards a lively video content presentation. IEEE Transactions on Multimedia, 14(3):858--870

2012
[24]

Hideaki Yanagisawa, Takuro Yamashita, and Hiroshi Watanabe. 2018. A study on object detection method from manga images using cnn. In 2018 International Workshop on Advanced Image Technology (IWAIT), pages 1--4. IEEE

2018
[25]

Xin Yang, Zongliang Ma, Letian Yu, Ying Cao, Baocai Yin, Xiaopeng Wei, Qiang Zhang, and Rynson WH Lau. 2021. Automatic comic generation with stylistic multi-page layouts and emotion-driven text balloon generation. ACM Transactions on Multimedia Computing, Communications, and A...

2021
[26]

Albin Zehe, Leonard Konle, Lea Katharina D \"u mpelmann, Evelyn Gius, Andreas Hotho, Fotis Jannidis, Lucas Kaufmann, Markus Krug, Frank Puppe, Nils Reiter, et al. 2021. Detecting scenes in fiction: A new segmentation task. In Proceedings of the 16th conference of the European ...

2021
[27]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Khaled Bayoudh, Raja Knani, Fay c al Hamdaoui, and Abdellatif Mtibaa. 2022. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, 38(8):2939--2970

work page 2022

[2] [2]

Neil Cohn. 2010. The limits of time and transitions: Challenges to theories of sequential image comprehension. Studies in Comics, 1(1):127--147

work page 2010

[3] [3]

Neil Cohn. 2011. A different kind of cultural frame: An analysis of panels in american comics and japanese manga. Image & Narrative, 12(1):120--134

work page 2011

[4] [4]

Neil Cohn. 2013. The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. Bloomsbury Academic, New York

work page 2013

[5] [5]

Alexander Dunst, Rita Hartel, and Jochen Laubrock. 2017. The graphic narrative corpus (gnc): design, annotation, and analysis for the digital humanities. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 3, pages 15--20. IEEE

work page 2017

[6] [6]

Arpita Dutta, Samit Biswas, and Amit Kumar Das. 2022. Bcbid: first bangla comic dataset and its applications. International Journal on Document Analysis and Recognition (IJDAR), 25(4):265--279

work page 2022

[7] [7]

Azuma Fujimoto, Toru Ogawa, Kazuyoshi Yamamoto, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2016. Manga109 dataset and creation of metadata. In Proceedings of the 1st international workshop on comics analysis, processing and understanding, pages 1--5

work page 2016

[8] [8]

Goran Glava s , Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation using semantic relatedness graphs. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 125--130. Association for Computational Linguistics

work page 2016