{"id":"09054bc2-3001-4785-9bb3-fe3f000be315","arxiv_id":"2501.09884","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A semi-supervised adaptation of narrative maps extracts visual storylines from historical photo archives and often beats random sampling on timelines of 10+ images, though the evaluation shares the expert who supplied labels.","lead":"This paper adapts the text-based narrative maps algorithm to extract storylines from collections of historical photographs, using a semi-supervised approach with expert labels. In a case study of 500 photos from a 1928 Bolivia expedition, the method outperforms random sampling on longer timelines, but the evaluation is partly guided by the same expert who created the ground truth.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed NM-over-random advantage may reflect the expert's own labels and dates being fed into NM while withheld from RS; independent ground truth is needed before accepting the effectiveness claim.","rationale":"The reader's weakest assumption identifies the same core problem, and I agree it is the load-bearing issue. The paper's strongest claim is inseparable from the evaluation: 'narrative maps approach generally outperforms random sampling for longer timelines (10+ images, p < 0.05), with expert evaluation confirming...' For that claim to be evidence of an effective extraction tool, the comparison must be against a ground truth that is not also the source of the algorithm's supervision. In the present design, the label-spreading step in Section 3.2 consumes thematic labels and date seeds from Section 3.5, the same expert who authorizes the six evaluation timelines. The extraction is also run with the expert's start/end images and K set to the expert's timeline length. Random sampling is not given any of this supervision, so the significant p-values in Table 1 could simply reflect that NM has access to expert chronology and RS does not. The qualitative evaluation is likewise performed by the same expert and is unblinded. These are not allegations of misconduct; they are structural features of the evaluation that make the central claim underdetermined. The paper is valuable as a proof of concept, and the authors are transparent about several limitations, but they do not test the independence of the ground truth. The concrete test I propose—a second expert who only evaluates and never supplies labels—directly isolates this confound. If it passes, the claim is much stronger; if it fails, the conditional verdict should be maintained or strengthened. Since the reader already recommended CONDITIONAL, no verdict change is needed.","tokens_in":10003,"tokens_out":4803,"duration_ms":50325,"concrete_test":"Hold the evaluator independent from the label provider: recruit a second historian with expertise on the Sacambaya expedition, give them no access to the labels/date seeds used in Section 3.2, and ask them to construct a 30-image timeline. Then rerun the Section 3.6 protocol for L=30 using the original expert's labels for NM, the new expert's timeline as ground truth, and the same 20 random shuffles for both NM and RS. If the NM-vs-RS advantage in DTW distance and cosine similarity is no longer significant at p<0.05, the reported superiority depends on the ground-truth author also supplying the algorithm's supervision. Repeating for L=10 and L=20 would confirm the pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that NM outperforms random sampling for timelines of length 10+ (Table 1). The load-bearing assumption is that the expert-curated timelines are an independent ground truth. That assumption fails in the current design: Sections 3.2 and 3.5 state that the same domain expert who built the six evaluation timelines also supplied the thematic labels and date seeds used by label spreading, and Section 3.6 fixes the start/end images and K to the expert's choices. The NM path therefore has direct access to the expert's chronological structure, while the RS baseline is given no such information, so the comparison conflates algorithmic narrative extraction with injected expert supervision. The problem is compounded by the qualitative evaluation: the expert who rates coherence and historical accuracy is the same person who supplied the labels and the reference timeline, and the evaluation is unblinded. The manuscript's own limitation paragraph (Section 5.1) acknowledges that expert-provided labels can introduce bias, but no experiment isolates this effect. Given that the discussion also notes the dataset is small and highly similar, even small DTW/cosine gaps (e.g., L=10 high-dim similarity 0.915 vs 0.895, p=0.0399) being significant is not enough to establish an intrinsic advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the text-based Narrative Maps algorithm to visual data for semi-supervised narrative extraction from historical photographs. Using DETR features on a 500-image subset of the ROGER collection, the method applies label spreading to propagate expert-provided thematic and date labels, builds a directed coherence graph, and extracts a main storyline between fixed start and end images. The extracted narratives are compared with six expert-curated timelines (lengths 5–30) using DTW distance and average cosine similarity, against a random sampling baseline, and a qualitative assessment by a domain expert is reported. The central claim is that the Narrative Maps approach generally outperforms random sampling for longer timelines (10+ images, p < 0.05), with expert evaluation confirming historical accuracy and coherence.","tokens_in":10170,"tokens_out":5429,"duration_ms":56868,"significance":"If the effectiveness claim were established, the paper would offer a useful proof-of-concept for computational narrative analysis in visual cultural heritage, and the release of source code and experiments is a practical strength. However, the evaluation design does not currently support the central claim: the same expert knowledge is used to construct the ground truth, to provide the labels and date seeds consumed by the method, and to judge the qualitative output. The comparison against random sampling therefore conflates algorithmic narrative extraction with injected expert supervision. The authors are transparent about several limitations, including a small and highly homogeneous dataset and the reliance on expert labels, but no experiment isolates the effect of this supervision. The significance of the reported quantitative results is additionally weakened by the lack of multiple-comparison control and by small effect sizes.","major_comments":[{"comment":"The ground truth and the supervision given to the proposed method come from the same expert. Section 3.5 states that a domain expert created the six evaluation timelines and that the expert's partial labels and dates are fed into the semi-supervised pipeline; Section 3.2 uses these expert-provided thematic labels and date approximations as seeds for label spreading; and Section 3.6 fixes the start and end images to the expert-curated timeline. The random sampling baseline receives none of this supervision. As a result, the NM-over-RS advantage reported in Table 1 may reflect injected expert knowledge rather than intrinsic narrative extraction ability. This is load-bearing for the paper's central claim. The manuscript's own limitation paragraph in Section 5.1 acknowledges that expert-provided labels can introduce bias, but no experiment isolates this effect. I would need to see either an independently constructed ground truth, an ablation that withholds the expert labels and dates from NM, or a baseline that is given the same cluster and date information before accepting the effectiveness claim.","section":"Sections 3.2, 3.5, 3.6"},{"comment":"The statistical comparison uses 20 trials and raw t-tests for each timeline length and each metric, with six lengths and four metrics resulting in 24 tests. No multiple-comparison correction is applied. With this many tests, several p < 0.05 results are expected by chance; for instance, after a Bonferroni correction the high-dimensional similarity differences at L=10 (p=0.0399), L=15 (p=0.0257), L=20 (p=0.0033), and L=30 (p=0.0087) would no longer meet the family-wise significance threshold. The effect sizes are also small (e.g., L=10 high-dimensional similarity 0.915 vs 0.895), and the L=25 high-dimensional similarity comparison is not significant (p=0.3514). The abstract's sweeping claim that NM generally outperforms random sampling for 10+ images is therefore stronger than the currently reported evidence supports.","section":"Section 4.1, Table 1"},{"comment":"The qualitative evaluation is performed by the same domain expert who created the baseline timelines and supplied the labels and dates used by the method, and there is no indication of blinding. This cannot serve as an independent confirmation of historical accuracy and coherence. In particular, the statement that the algorithmic selections could sometimes be interpreted as possessing greater relevance from an objective, image-content-based perspective is itself a subjective judgment by the same expert. A blind evaluation by a second expert, or at minimum a clearly independent evaluator, is needed to support the qualitative claims.","section":"Section 4.2"},{"comment":"The authors note that the dataset is small and highly similar, that random sampling performs surprisingly well in the high-dimensional space, and that both NM and RS tend to emulate the underlying class distribution. These observations are consistent with the small absolute differences in Table 1 and suggest that the reported advantages may partly reflect distributional properties of the collection rather than narrative structure. To contextualize the NM advantage, the paper should report a stronger baseline that is matched in information access, such as stratified random sampling by the expert's own thematic clusters, or a distribution-matching baseline. Without such a baseline, the comparison against uniform random sampling is too weak to establish that the method extracts meaningful storylines rather than merely reproducing the cluster distribution.","section":"Section 5"}],"minor_comments":[{"comment":"The word 'Departament' in the affiliation for the first author should be corrected to 'Department'.","section":"Author affiliation"},{"comment":"It is ambiguous whether the 'domain experts' in Section 3.2 and the 'domain expert' in Section 3.5 are the same individual; this should be stated explicitly because the independence of the ground truth and the supervision depends on it.","section":"Sections 3.2 and 3.5"},{"comment":"The text says 'we do not explicitly evaluate coherence' and then immediately reports that 'average coherence values of the extracted NM are higher compared to the random sampling,' which is confusing; please clarify whether coherence is a reported metric or not.","section":"Section 4.1"},{"comment":"The caption refers to the 'unsupervised narrative maps algorithm,' but the paper's method is semi-supervised; the wording should be aligned.","section":"Figure 3 caption"},{"comment":"The description of the t-test analysis does not state whether the tests are paired or unpaired, nor whether the 20 trials are independent; adding this information would improve reproducibility.","section":"Section 3.6"},{"comment":"Reference [23] contains a typo: 'Darthmouth College' should be 'Dartmouth College.'","section":"Reference [23]"}],"recommendation":"major_revision","confidential_remarks":"The core issue is not a technical error in the algorithm but a load-bearing evaluation-design confound. The authors are transparent about their reliance on expert labels and release their code, which are strengths. I would encourage the editor to request a revised version that includes an independent or blinded qualitative evaluation, an ablation or an information-matched baseline that isolates the effect of the expert-provided labels and dates, and appropriate multiple-comparison control. With those additions, the paper's proof-of-concept claim could become defensible; as it stands, the abstract's effectiveness claim exceeds what the evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate and clearly written proof-of-concept for adapting narrative maps to visual data, but the central effectiveness claim is not yet supported because the same expert knowledge is fed into the algorithm and used to build the ground truth. The stress-test concern lands.\n\nWhat's actually new: the semi-supervised modification, replacing the unsupervised clustering step with label spreading on DETR embeddings, and the application to the ROGER historical photograph collection. That's a modest but real adaptation, and the paper gives a reproducible pipeline description. Credit where due: the authors are transparent about several limitations, including the reliance on expert labels, the small homogeneous dataset, and the fact that random sampling does surprisingly well. They also note that high-dim similarity gains are marginal after multiple-comparison adjustment.\n\nSoft spots, in order of seriousness. First, the evaluation confound: the same domain expert supplied the thematic labels and date seeds used by label spreading, created the six ground-truth timelines, and later qualitatively rated the algorithm's output. Start and end images are fixed for both NM and RS, but NM receives the expert's chronological and thematic structure while RS receives none of it. So the reported p-values for L>=10 likely measure injected supervision as much as narrative extraction. The paper's own limitation paragraph acknowledges labels can bias results, but no experiment isolates that effect. Second, the statistical reporting is raw t-tests with no multiple-comparison correction; for the high-dimensional similarity metric, significance largely disappears after correction by their own account. Third, reproducibility details are missing: the GitHub link is absent and several parameters (K, mincover, label-spreading settings, UMAP) are left unspecified. None of this makes the approach worthless, but it does mean the paper currently supports a proof-of-concept, not the stronger claim in the abstract.\n\nWho this is for: digital humanities researchers and anyone building tools for large photo archives. A serious referee should see it, with the expectation that the authors add an independent ground-truth evaluator, an unsupervised narrative maps baseline, a no-expert condition, and corrected statistics. As it stands, I would not cite the effectiveness claim, but I would cite the adaptation idea if I were working on visual narrative extraction.","headline":"A legitimate, clearly written proof-of-concept for adapting narrative maps to visual data, but the effectiveness claim is undercut by an evaluation that feeds the expert's own labels and dates into the algorithm while withholding them from the random baseline.","tokens_in":10754,"tokens_out":2040,"would_cite":false,"duration_ms":21286,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A semi-supervised algorithm extracts coherent visual storylines from historical photo collections.","keywords":["visual narrative extraction","semi-supervised learning","narrative maps","historical photographs","cultural heritage","DETR","label spreading","Sacambaya Expedition"],"falsifier":"Conduct the comparison with a second expert who is blinded to the first expert's labels and independently builds the ground-truth timelines and seed labels; if the narrative-maps advantage over random sampling disappears or becomes non-significant for 10+ image timelines, the reported performance depended on shared expert knowledge rather than intrinsic narrative extraction.","tokens_in":9734,"feed_emoji":"📷","tokens_out":6564,"duration_ms":57884,"temperature":0.7,"pith_summary":"The paper claims that a semi-supervised adaptation of the narrative maps algorithm can extract coherent visual storylines from historical photographs. Applied to 500 images from the 1928 Sacambaya Expedition, the extracted narratives align with expert-curated timelines better than random sampling for timelines of 10 or more images, with most differences reaching statistical significance. An expert's qualitative review of one extracted storyline judged it coherent, relevant, and historically accurate. If correct, this gives historians and archivists a route from large, partially labeled photo collections to readable narrative sequences.","feed_headline":"Semi-supervised method finds visual narratives in 1928 photo archive","feed_subtitle":"The method beats random sampling on timelines of 10+ images and aligns with an expert's historical account","key_machinery":"The central object is the adapted narrative maps pipeline: a directed acyclic graph whose nodes are photographs and whose edge weights encode coherence, computed as a combination of DETR feature similarity and label-spreading outputs for thematic category and date. A linear program maximizes the weakest-link coherence of the path from a fixed source to a fixed target while enforcing coverage constraints over thematic clusters, and the main storyline is the maximum-likelihood path through that graph. The key adaptation is the replacement of the original unsupervised clustering with semi-supervised label spreading, which is what allows partial expert labels and date seeds to steer the narrative structure.","core_discovery":"The paper's central claim is that the narrative maps algorithm, originally designed for news text, can be adapted to visual data by replacing its unsupervised clustering step with semi-supervised label spreading and by defining edge coherence from DETR image features combined with propagated thematic and temporal labels. The extracted narrative is the most coherent directed path from a fixed start image to a fixed end image, chosen by a linear program that maximizes the weakest-link coherence subject to coverage constraints over thematic clusters. Against expert-curated timelines of lengths 5 through 30, the method significantly outperforms random sampling on DTW distance and cosine similarity for timelines of 10 or more images in the UMAP-reduced space, and for most longer timelines in the original high-dimensional space. The authors interpret this as evidence that the coherence-based graph objective captures narrative structure rather than merely following the dataset's class distribution, while acknowledging that random sampling also performs well on the shortest timeline and on this small, visually homogeneous collection.","pith_inferences":["If the single-expert ground truth is the main confound, a decisive follow-up is to have a second, independent expert construct timelines and seed labels from the primary texts alone; a large performance drop would indicate the algorithm is partly recovering that annotator's own labeling rather than intrinsic narrative structure.","The same pipeline could transfer to other partially ordered visual corpora, such as press photo archives or personal albums, where the narrative is retrospective rather than authored.","Because only partial labels and a start/end pair are needed, the method could become an interactive exploration tool that proposes the most coherent storyline between two chosen images and lets a researcher edit and rerun.","Domain-specific fine-tuning of the visual encoder on historical photography, rather than generic object-detection features, is a testable improvement the paper leaves open."],"forward_implications":["For collections with partial expert annotations, the method produces a narrative timeline without requiring every image to be labeled.","The advantage over random sampling is clearest for timelines of 10 or more images, so the method is best suited to reconstructing extended event sequences rather than very short stories.","Using UMAP-reduced embeddings preserves the advantage while lowering computational cost, though the paper reports mixed results in the high-dimensional space for some timeline lengths.","The extracted narratives are not exact replicas of an expert's selection; they offer alternative, verifiable orderings that keep human oversight in the loop.","The authors expect the benefit over random sampling to be larger in noisier, more diverse collections, where random selections are less likely to land on coherent sequences."],"supporting_citations":[{"why":"Supplies the base narrative maps algorithm, including the graph representation, coherence maximization, and linear programming framework that the paper adapts to images.","marker":"[13]"},{"why":"Provides the pre-trained DETR model used to extract the visual feature vectors that drive image similarity and coherence computation.","marker":"[4]"},{"why":"Supplies the label-spreading algorithm used to propagate thematic category and date labels from partial expert annotations.","marker":"[30]"},{"why":"Introduces the ROGER dataset and prior work on narrative extraction from these images, which the current study extends.","marker":"[18]"},{"why":"Defines the 'weakest link' coherence principle that the narrative maps objective is built on, linking story coherence to the weakest connection in the path.","marker":"[26]"},{"why":"Provides UMAP dimensionality reduction, used to build the low-dimensional embedding space for one of the evaluation conditions.","marker":"[19]"},{"why":"Defines dynamic time warping, the alignment metric used to compare extracted narratives with expert-curated timelines.","marker":"[20]"},{"why":"Primary textual source that the expert used to establish the expedition chronology underlying the baseline timelines.","marker":"[12]"},{"why":"Secondary historical systematization, supported by contemporary newspaper accounts, that the expert used to construct the baseline timelines.","marker":"[21]"}],"fun_headline_variants":["Semi-supervised AI extracts narratives from 1928 expedition photos","Algorithm finds storylines in 1928 Bolivia photo archive","Deep learning decodes 1928 photo archive into narratives","AI narrates 1928 Bolivia expedition from photos","Semi-supervised method outperforms random sampling on photo stories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expert-curated timelines serve as ground truth, yet the same expert supplied the thematic labels and date seeds used by the algorithm, and the start and end images are fixed to that expert's choices; if this ground truth is not independent, the advantage over random sampling may largely reflect injected expert knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Semi-supervised AI extracts narratives from 1928 expedition photos","Algorithm finds storylines in 1928 Bolivia photo archive","Deep learning decodes 1928 photo archive into narratives","AI narrates 1928 Bolivia expedition from photos","Semi-supervised method outperforms random sampling on photo stories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001443,"raw_usage":{"total_tokens":5831,"prompt_tokens":978,"completion_tokens":4853,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":4770}},"tokens_in":594,"tokens_out":4853,"duration_ms":34929,"temperature":1.0,"reasoning_tokens":4770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:33:59.022389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct the comparison with a second expert who is blinded to the first expert's labels and independently builds the ground-truth timelines and seed labels; if the narrative-maps advantage over random sampling disappears or becomes non-significant for 10+ image timelines, the reported performance depended on shared expert knowledge rather than intrinsic narrative extraction.","supporting_citations":[{"cited_title":"Proceedings of the ACM on Human- Computer Interaction 4(CSCW3), 1–33 (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the base narrative maps algorithm, including the graph representation, coherence maximization, and linear programming framework that the paper adapts to images."},{"cited_title":"Advances in neural information processing systems16 (2003)","cited_arxiv_id":null,"evidence_quote":"Supplies the label-spreading algorithm used to propagate thematic category and date labels from partial expert annotations."},{"cited_title":"In: Text2Story@ ECIR","cited_arxiv_id":null,"evidence_quote":"Introduces the ROGER dataset and prior work on narrative extraction from these images, which the current study extends."},{"cited_title":"In: Proceed- ings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining","cited_arxiv_id":null,"evidence_quote":"Defines the 'weakest link' coherence principle that the narrative maps objective is built on, linking story coherence to the weakest connection in the path."},{"cited_title":"John Long, London (1934)","cited_arxiv_id":null,"evidence_quote":"Primary textual source that the expert used to establish the expedition chronology underlying the baseline timelines."},{"cited_title":"Robert Gerstmann y el tesoro de Sacambaya","cited_arxiv_id":null,"evidence_quote":"Secondary historical systematization, supported by contemporary newspaper accounts, that the expert used to construct the baseline timelines."}],"review_version":1}