{"id":"94ac60b4-76c8-4c10-9578-b78bc1a26461","arxiv_id":"2501.05480","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A supervised authorship-verification system attributes the disputed Latin treatise 'Questio de aqua et terra' to Dante Alighieri with very high confidence, aided by a new application of the oversampling technique DRO.","lead":"A computational stylometry study compared 330 medieval Latin texts and found that Dante is the most likely author of the disputed 'Questio de aqua et terra', a treatise long questioned by philologists. The technique that mattered most, a class-balancing method called DRO, lifted verification accuracy from F1 0.400 to 0.970.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attribution claim depends on the closed-world assumption that the non-Dantean alternative is represented in the corpus; the paper's own 'Adso da Melk' scenario (Section 6) concedes this may fail, and LOO testing does not measure rejection of an unseen author.","rationale":"The reader's weakest assumption—the closed-world comparator set—is exactly the load-bearing point. The paper's Section 6 explicitly raises the Adso da Melk scenario and concedes that adding an unseen author could overturn the verdict, but then conjectures it is unlikely. That conjecture is the crux: the LOO experiment does not simulate an unseen author, because every held-out document still has other documents by the same author in training. The AA experiment in Section 5.1 also operates within a closed set of 38 authors and cannot detect an author outside it. Thus the probability 0.999999967 is conditional on the model's training distribution, not a statement about the open world. The reported F1=0.970 and soft F1=0.900 do not support the extreme posterior, a separate overstatement, but the closed-world concern is more fundamental because it bears on the truth of the attribution, not just its confidence. A leave-one-author-out test would empirically estimate the false-acceptance rate for unseen authors and would settle whether the concern lands. The reader's CONDITIONAL verdict is appropriate: the study is valuable and reproducible, but the central claim requires an additional open-set evaluation and a carefully hedged statement. No change to the reader's verdict is needed.","tokens_in":17219,"tokens_out":5892,"duration_ms":58455,"concrete_test":"Perform a leave-one-author-out evaluation: for each of the 14 non-Dantean authors in MedLatinQuestio with at least two texts, remove all texts by that author from the training set, train the maximally accurate binary verifier (logistic regression + five feature sets + DRO, as in Section 4) on Dante versus all remaining non-Dantean authors, and then test the held-out author's full texts. Count how many are falsely classified as Dantean. If the false-acceptance rate is materially above the LOO false-positive rate (1/314), the comparator set is not representative of unseen non-Dantean authors, and the Questio verdict cannot be stated as an open-world conclusion. Repeating with DRO disabled would also reveal whether oversampling inflates this risk.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5, that Dante is the true author with Pr(Dante|Questio)=0.999999967, rests on the closed-world premise that the alternative to Dante is represented among the 38 authors in MedLatinQuestio. The LOO protocol in Section 3.4 leaves out one document while all other documents by the same author remain in training, so it measures recognition of a known style, not rejection of an author entirely absent from the training set. An unknown forger ('Adso da Melk,' Section 6) is a new class the verifier has never seen; the posterior probability is computed under the model's closed-world class distribution and is not an open-world estimate of the chance that the Questio is non-Dantean. The paper explicitly acknowledges this scenario but dismisses it with the conjecture that character n-grams are 'difficult, if not impossible, to imitate'—an empirical claim offered without evidence. Because the entire attribution hinges on the comparator set fairly covering plausible non-Dantean authors, and because the paper itself concedes the set may be incomplete, the reported experiments do not establish the open-world claim. This is a limitation of evidence, not an internal inconsistency; it could be addressed by an explicit open-set evaluation and a more hedged conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a computational authorship verification study of the disputed treatise 'Questio de aqua et terra', attributed by tradition to Dante Alighieri. The authors assemble a corpus of 330 medieval Latin texts (MedLatinQuestio), train a binary classifier (logistic regression with hand-crafted stylometric feature sets) to distinguish Dante from non-Dante, and report a leave-one-out F1 of 0.970. They attribute the high performance mainly to Distributional Random Oversampling (DRO), which is used here for the first time in authorship verification. When the trained verifier is applied to the Questio, it returns a posterior probability Pr(Dante|Questio)=0.999999967, which the authors interpret as strong evidence for Dantean authorship. Additional authorship attribution experiments on 38 candidate authors also rank Dante first, with a probability of 0.737. The authors make their code and dataset publicly available.","tokens_in":17451,"tokens_out":4243,"duration_ms":40012,"significance":"If the attribution result holds, this work would provide a strong quantitative contribution to a long-standing philological debate and would showcase DRO as a useful technique for authorship verification in low-resource historical corpora. The paper is carefully structured, the corpus is a valuable resource, and the experiments are described in sufficient detail to be reproduced. The main strength is the public release of code and data, and the explicit consideration of topic/genre confounding in feature selection. However, the central claim of near-certain Dantean authorship rests on a closed-world assumption that the paper itself partially acknowledges; this limits the significance of the strong posterior probability until an open-set evaluation is provided.","major_comments":[{"comment":"The claim that Pr(Dante|Questio)=0.999999967 means '33 chances in a billion' that the Questio is non-Dantean is an overinterpretation of a model posterior. The logistic regression posterior is calibrated only on average over the 330-text corpus (F_s1=0.900); there is no evidence that probabilities at the 1e-9 level are meaningful. The paper should either report the raw confidence and explicitly state that extreme posterior values are not validated, or temper the wording in the abstract and Section 5.","section":"Section 5 and Section 6"},{"comment":"The leave-one-out protocol leaves out one document while other texts by the same author remain in the training set, so it measures closed-set recognition of known authors, not rejection of an author entirely absent from the training set. The 'Adso da Melk' scenario in Section 6 concedes that a real non-Dantean author outside the 38-author set could, in principle, lead the verifier to a different outcome. The conclusion that Dante is the true author with near-certainty is therefore not supported by the current LOO evidence; an explicit open-set evaluation (e.g., training on a subset of authors and testing on held-out authors) is needed to substantiate the open-world claim.","section":"Section 3.4 and Section 4"},{"comment":"The reported F1=0.970 is computed on the same 330 texts that were used to guide the greedy ablation and to identify the '10 hardest texts' on which the final configuration was selected. This makes the reported accuracy an optimistic, selection-dependent estimate rather than an unbiased estimate of the deployed verifier's performance. The authors should either use nested cross-validation or explicitly describe the reported F1 as a model-selection result, not an independent accuracy estimate.","section":"Section 4, Table 2 and ablation procedure"}],"minor_comments":[{"comment":"Typo: 'for the agoals of our task' should read 'for the goals of our task'.","section":"Section 3.1"},{"comment":"Typo: 'morphosyntactyic verbal endings' should read 'morphosyntactic verbal endings'.","section":"Section 3.3"},{"comment":"The vanilla accuracy value is reported as 0.996 in Section 6, while Section 4 computes it as 329/330=0.997. These values should be made consistent.","section":"Section 6"},{"comment":"The caption states that the 3rd column indicates whether the inference is correct and the 4th column the confidence, but the table appears to have only three columns, with the ✓/✗ symbol placed inside the 'Text' column. The caption should be updated to match the actual table layout.","section":"Table 1 caption"},{"comment":"The Kestemont (2014) reference gives the workshop year as 'CLfL 2024'; the year should be 2014 to match the publication date.","section":"Reference list"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is internally consistent and the experimental work is careful, but the conclusions go beyond what the closed-set evaluation can support. The authors' own Section 6 caveat is more honest than the abstract's 'highly confident prediction'. I would ask for a revised version that either adds an open-set evaluation or clearly restricts the conclusion to the closed-world setting, and that avoids interpreting a raw logistic-regression posterior as a calibrated probability at the 1e-9 level."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time, but read it with the headline claim in brackets. The paper does three things well: it builds a new medieval Latin corpus (MedLatinQuestio), it applies Distributional Random Oversampling to authorship verification for the first time, and it publishes code and data so the results can be checked. The LOO F1=0.970 on 330 heterogeneous texts is solid, and the ablation showing DRO lifts F1 from 0.400 to 0.970 is genuinely striking. The AA cross-check, with Dante ranking first among 38 authors, is a useful second opinion, not a rhetorical add-on.\n\nThe soft spots are real and tend to cluster around the strength of the final attribution. First, the closed-world premise: LOO removes one document but keeps other texts by the same author in training, so it measures recognition of a known style, not rejection of an unseen author. The paper's own \"Adso da Melk\" passage concedes this, then dismisses it with an empirical claim — that character n-grams are effectively impossible to imitate — that is asserted, not demonstrated. That leaves Pr(Dante|Questio)=0.999999967 as a closed-world object, not an open-world probability that the text is non-Dantean. Second, the reported F1 is not an unbiased estimate of the final system, because the same corpus was used to select the configuration (the \"10 hardest texts\" guided the feature ablation). A nested validation would fix this. Third, the extreme posterior is not obviously calibrated: the system's soft F1 is 0.900, so its confidence scores are only moderately reliable on average, yet the Questio receives a probability with nine nines. Reporting that number without a calibration caveat oversells the result.\n\nThese are fixable problems, not fatal ones. The paper is honest about many of its limitations, the corpus and code are public, and the DRO self-citation is legitimate because the method is theirs and it is evaluated on held-out texts. The right outcome is a revision, not a rejection: add a nested validation, run an explicit open-set or unseen-author experiment, and hedge the conclusion so that it says \"consistent with Dantean authorship under a closed-world model\" rather than \"true author.\" I would send this to a serious natural-language-processing or digital-humanities venue, and I would bring it to a reading group myself.","headline":"A serious, transparent computational effort with a genuine methodological contribution, but the headline probability for Dante's authorship is not supported by the evidence once the closed-world assumption and the model-selection protocol are examined.","tokens_in":18011,"tokens_out":2040,"would_cite":true,"duration_ms":22698,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A computational authorship-verification study attributes the disputed 'Questio de aqua et terra' to Dante with posterior probability 0.999999967.","keywords":["authorship verification","Questio de aqua et terra","Dante Alighieri","stylometry","distributional random oversampling","medieval Latin","logistic regression","cultural heritage"],"falsifier":"The closed-world premise would be falsified if adding even a small set of Latin works by a previously excluded fourteenth-century cosmologist to the MedLatinQuestio corpus and rerunning the leave-one-out protocol caused the verifier to reclassify the Questio as non-Dantean or materially lowered its confidence.","tokens_in":16970,"feed_emoji":"📜","tokens_out":11700,"duration_ms":97052,"temperature":0.7,"pith_summary":"The paper asks whether the Latin cosmological treatise 'Questio de aqua et terra' was written by Dante Alighieri, a question that has divided scholars because the work's cosmology seems to clash with Dante's other writings and no contemporary manuscript survives. To answer it, the authors assemble a corpus of 330 thirteenth- and fourteenth-century Latin texts, train authorship verifiers that combine logistic regression with stylometric features, and evaluate them by leave-one-out cross-validation. Their best verifier is correct on 329 of the 330 texts ($F_1=0.970$), and the main reason for that accuracy is Distributional Random Oversampling (DRO), a technique that synthesizes training examples for the scarce Dante class and lifts $F_1$ from 0.400 to 0.970. Applied to the Questio, the verifier attributes the text to Dante with posterior probability $\\Pr(\\mathrm{Dante}\\mid\\mathrm{Questio})=0.999999967$. The paper's wider point is that DRO makes computational authorship verification practical for cultural-heritage texts, where only a handful of texts by the candidate author usually survive.","feed_headline":"Dante authorship of the 'Questio' affirmed at 0.999999967","feed_subtitle":"Leave-one-out tests on 330 medieval Latin texts hit F1=0.970, and the disputed cosmology treatise lands on Dante.","key_machinery":"The load-bearing mechanism is Distributional Random Oversampling (DRO), a technique that generates synthetic training examples for the minority class by appending random 'latent' features to each text vector, exploiting the distributional hypothesis that words—and, here, syntactic and character-level units—with similar distributions serve similar functions. DRO is used for the first time in authorship verification, and ablating it drops $F_1$ from 0.970 to 0.400; removing character n-grams drops it to 0.455. The supporting machinery is the MedLatinQuestio corpus of 330 medieval Latin texts by 38 authors, segmented into 5,430 training instances, with the final configuration using 12,807 natural features plus 21,770 DRO-generated latent features at a 20/80 positive/negative ratio.","core_discovery":"The paper's central claim is that the Questio is Dantean and that computational verification can establish this with near-certainty. On the authors' own evaluation, a logistic-regression verifier using token lengths, function words, sentence lengths, POS n-grams, and character n-grams, supplemented by DRO-generated synthetic examples, makes 329 correct decisions out of 330 in leave-one-out tests on the MedLatinQuestio corpus; the sole error is Boccaccio's Epistle 23, misclassified as Dantean. Trained on all 330 texts and applied to the Questio, the verifier reports $\\Pr(\\mathrm{Dante}\\mid\\mathrm{Questio})=0.999999967$, equivalent to about 33 chances in a billion that the text is non-Dantean. An authorship-attribution system trained on the same corpus ranks Dante first among 38 authors at probability 0.737, with the nearest competitor at 0.067. The authors read this combination, together with the near-impossibility of consciously imitating character n-gram frequencies, as strong evidence against the forgery hypothesis.","pith_inferences":["The paper's 'Adso da Melk' caveat is left qualitative: the 0.999999967 probability is conditional on the true author being one of the 38 comparators, so adding a genuine non-Dantean author to the corpus could in principle flip the verdict even if the current estimate is stable.","The authors used DRO on non-content features and found it works, which suggests the distributional hypothesis operates below the word level; a direct test would ablate DRO per feature type to see which levels contribute most.","A natural extension is to run the same verifier on the 1509 printed edition of the Questio and on the disputed Pietro Alighieri commentary that mentions it; textual variants there could change the feature vectors that drive the attribution."],"forward_implications":["If the verifier's result is correct, the Questio joins Dante's Latin corpus, and the philological arguments from cosmology and from the absence of contemporary references need to be squared with a Dantean attribution.","DRO's contribution implies that authorship verification on imbalanced cultural-heritage corpora can be substantially improved by distributional oversampling, not just by better feature engineering.","The feature ablation results indicate that character n-grams and DRO-generated latent features carry most of the authorship signal; future verifiers for medieval Latin should treat those as core rather than optional.","The MedLatinQuestio corpus and code, released with the paper, let other researchers reproduce the $F_1=0.970$ result and apply the same pipeline to other disputed texts."],"supporting_citations":[{"why":"Introduces Distributional Random Oversampling, the technique whose ablation is the largest single contributor to the verifier's accuracy.","marker":"Moreo et al., 2016"},{"why":"Supplies the MedLatinEpi and MedLatinLit datasets that make up most of the MedLatinQuestio corpus, including the Dantean treatises and twelve Dante epistles.","marker":"Corbara et al., 2022"},{"why":"Defines the character-masking features that enter the candidate feature pool and survive into the final configuration's ablations.","marker":"Stamatatos, 2018"},{"why":"Provides the distributional hypothesis that DRO operationalizes when it generates latent distributional features.","marker":"Lenci and Sahlgren, 2023"},{"why":"Documents the ineffectiveness of syllabic-quantity features on medieval Latin, which the paper cites when excluding them from the pool.","marker":"Corbara et al., 2023"},{"why":"Gives the philological argument that the Questio's cosmology postdates Dante, the historical doubt the verifier is answering.","marker":"Fioravanti, 2017"},{"why":"Presents the recent philological case against authenticity and prompts the computational investigation of the Questio.","marker":"Casadei, 2025"}],"fun_headline_variants":["Questio is Dante's: AI says 0.999999967","New AI method pins 'Questio' on Dante","Medieval corpus + DRO: Dante wins 'Questio' debate","F1=0.970, 33-in-a-billion odds: Dante wrote 'Questio'","Computational stylometry confirms Dante for 'Questio'"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true author of the Questio, if not Dante, is one of the 38 authors in the reference corpus; if the real author lies outside that closed set, the verifier's near-certain probability does not measure anything about the true authorship.","fun_headline_variants_meta":{"raw":{"variants":["Questio is Dante's: AI says 0.999999967","New AI method pins 'Questio' on Dante","Medieval corpus + DRO: Dante wins 'Questio' debate","F1=0.970, 33-in-a-billion odds: Dante wrote 'Questio'","Computational stylometry confirms Dante for 'Questio'"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1914,"prompt_tokens":991,"completion_tokens":923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":827}},"tokens_in":607,"tokens_out":923,"duration_ms":8695,"temperature":1.0,"reasoning_tokens":827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:42:24.194662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The closed-world premise would be falsified if adding even a small set of Latin works by a previously excluded fourteenth-century cosmologist to the MedLatinQuestio corpus and rerunning the leave-one-out protocol caused the verifier to reclassify the Questio as non-Dantean or materially lowered its confidence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Distributional Random Oversampling, the technique whose ablation is the largest single contributor to the verifier's accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MedLatinEpi and MedLatinLit datasets that make up most of the MedLatinQuestio corpus, including the Dantean treatises and twelve Dante epistles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the character-masking features that enter the candidate feature pool and survive into the final configuration's ablations."},{"cited_title":"and Sahlgren, M","cited_arxiv_id":null,"evidence_quote":"Provides the distributional hypothesis that DRO operationalizes when it generates latent distributional features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the ineffectiveness of syllabic-quantity features on medieval Latin, which the paper cites when excluding them from the pool."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the philological argument that the Questio's cosmology postdates Dante, the historical doubt the verifier is answering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the recent philological case against authenticity and prompts the computational investigation of the Questio."}],"review_version":1}