Fake or Real: The Impostor Hunt in Texts for Space Operations

Agata Kaczmarek; Artur Janicki; Dawid P{\l}udowski; Evridiki Ntagiou; Jakub Nalepa; Krzysztof Kotowski; Piotr Wilczy\'nski; Przemys{\l}aw Biecek; Ramez Shendy

REVIEW 4 major objections 5 minor 21 references

Fake or Real: The Impostor Hunt in Texts for Space Operations

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The central claim of this paper is that telling apart paired summaries from healthy and maliciously altered LLMs forms a competition benchmark for two real AI security threats in space operations.

desk verdict A well-organized competition announcement with a genuinely novel task design, but no results and an untested feasibility premise, so as a research paper there is nothing to referee yet. read the letter →

arxiv 2507.13508 v3 pith:IA7E7XQD submitted 2025-07-17 cs.LG cs.CR

Agata Kaczmarek , Dawid P{l}udowski , Piotr Wilczyński , Krzysztof Kotowski , Ramez Shendy , Evridiki Ntagiou , Jakub Nalepa , Artur Janicki

show 1 more author

Przemys{l}aw Biecek

This is my paper

classification cs.LGcs.CR

keywords SecureAIDataPoisoningLargeLanguageModelsSpaceOperationstextmanipulationdetectionblack-boxLLMauditpairedclassificationoverreliance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents a crowdsourced detection challenge, 'Fake or Real: The Impostor Hunt in Texts', built around pairs of LLM-generated summaries of astronomy articles. One summary in each pair comes from a correctly functioning model; the other comes from the same model after some undisclosed malicious modification, and participants must say which is which using only the texts, without access to the model. The organisers argue that this task operationalizes two security threats facing LLM-based assistants in space operations: data poisoning, which can make a model produce biased or false content, and overreliance, in which human operators trust such output without checking. If the competition succeeds, it would provide a public benchmark and community-validated detectors for manipulated LLM text that transfer beyond the space domain. The paper is a challenge description and reports no detection results yet.

What carries the argument

The load-bearing object is the paired-summary construction: for one source article, two summaries are produced by one LLM in a proper configuration and a tampered configuration, and the label is which text the proper model wrote. This pairing removes the usual task of distinguishing AI text from human text and turns the problem into a two-way forced choice between outputs of the same model, making black-box detection possible in principle. The attack recipes and the list of models are withheld until the challenge ends, so participants cannot exploit known artifacts; the provided baselines instead use word-level language detection and the ratio of Latin characters, which catch only crude distortions such as off-topic rambling or inserted Unicode signs.

What would settle it

After the hidden test labels are released, compute the winning solution's accuracy separately on pairs whose tampered summary is coherent, on-topic, and free of foreign words or odd characters; if accuracy on that subset is at or near 50 percent, the central assumption that tampering is textually detectable fails, and the benchmark would be either trivial on easy pairs or unsolvable on hard ones.

Watch

Extended reading notes

Core claim

On the paper's own terms, the claim is that distinguishing between the output of a benign LLM and the output of its maliciously modified version is a meaningful and solvable research task, and that solving it addresses data poisoning and overreliance. Each text pair is generated by the same underlying LLM, run once with the authors' proposed parameters and once after a deliberate manipulation; the participant sees only the two summaries, not the model or the attack. The score is simply the fraction of pairs correctly labelled. The paper also claims the resulting methods are universal, because detection relies on signals in the generated text itself rather than on topic-specific knowledge, even though the source articles come from the space domain. A deliberately broad manipulation list, including but not limited to hallucination-style corruption, is included precisely so that simple surface filters do not solve the whole benchmark.

Load-bearing premise

The whole setup assumes that maliciously tampering with an LLM leaves dependable text-level traces, so that a person or algorithm given only the two summaries can identify the altered one better than chance.

Editorial extensions

If this is right

A detector that wins the challenge would work in the realistic black-box setting where mission operators cannot inspect a deployed LLM's internals or training data.
Because both summaries describe the same article, a winning method must detect the fingerprint of tampering rather than generic differences in topic or style, which should transfer to other text domains.
If the benchmark works as intended, it gives space-operations projects a standard evaluation task for verifying LLM-based assistants before qualification.
The deliberately hidden attack list pushes solutions toward methods that catch unknown manipulations, not only the ones documented in the training set.
An accuracy metric over pairs makes results easy to compare across teams and provides a concrete leaderboard objective for a community contest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would be to rerun the winning solutions after the attack recipes are published, converting the one-off contest into a reusable test suite; the paper itself only commits to a summary report and an eventual workshop.
The organisers' own examples show that some tampered texts are detectable by a keyword or Unicode filter, so the final leaderboard may need to be split by attack difficulty to show whether top solutions generalize beyond simple artifacts.
If on-topic, coherent tampering produces no reliable text-level signal, then the no-model-access design would force detectors toward statistical style analysis, and performance on such pairs would collapse; this is testable once the test set is revealed.
The paired design could be transplanted to other safety-critical assistants, such as systems that generate operational procedures or telemetry annotations, where a tampered model could produce plausible but hazardous instructions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is not a research paper; it's a competition announcement, and there is no result to weigh yet. The paper describes a Kaggle challenge where participants must tell which of two LLM summaries of space-related articles comes from a benign model and which from a deliberately tampered one. Both summaries are AI-generated, and participants never get model access. That pairing is genuinely different from earlier detection competitions like LLM-Detect or the Trojan Detection Challenge, and it maps to a real operational concern in ESA's assurance work. The write-up is clear and honest: the authors state why they withhold attack definitions, give two simple baselines, specify the metric, and lay out the schedule. For a competition call, this is competent.

The problem is the load-bearing premise. The task is only worth benchmarking if real/fake pairs are separable without model access and not trivially separable. The paper never tests this. Worse, it concedes in Section 1.5 that some pairs can be solved by filtering keywords or Unicode characters, and the provided baselines do exactly that. So the organizers themselves acknowledge an unknown mixture of trivial and hard cases, but they give no distribution, no pilot accuracy, no human performance, not even a baseline score on the training data. The claim that methods will be 'universal' and applicable beyond space text is speculation at this point. There is no falsifiable claim to adjudicate.

That said, this isn't a flaw in execution; it's a mismatch with the form. The paper is a pre-registration, not a results paper. If Kaggle competitions are the object of study, a description like this could be publishable as a dataset/benchmark paper or a competition report. But as submitted to a research venue, there is simply nothing for a referee to check. My recommendation: desk-reject as a research paper, and invite a resubmission with results once the competition closes. A serious referee would have no evidence to evaluate otherwise.

Who gets value from this? People organizing security competitions might want to see the setup; researchers working on LLM output detection might find the dataset useful later. I wouldn't cite it until the results come in.

Referee Report

4 major / 5 minor

Summary. The paper describes the 'Fake or Real: The Impostor Hunt in Texts' Kaggle competition, organized as part of the European Space Agency's 'Assurance for Space Domain AI Applications' project. Participants are given pairs of LLM-generated summaries of articles from ESO's The Messenger; one summary in each pair is produced by a correctly functioning LLM and the other by an LLM that has been maliciously modified. The task is to identify which summary is the real one. The paper motivates the competition by two AI security concerns, data poisoning and overreliance, and presents the data-creation process, the accuracy metric, two simple baselines, the competition protocol, and early engagement statistics. No experimental results are reported: the paper contains no baseline accuracy, no leaderboard scores, no human performance, and no analysis of whether the task is solvable above chance. The attack definitions are explicitly withheld in Section 1.4, and the paper's schedule shows the competition was still ongoing at the time of writing.

Significance. If the competition works as intended, it would provide a novel benchmark for detecting manipulated LLM output in a paired 'AI-generated vs AI-generated' setting, which is relatively unexplored compared to AI-vs-human text detection. The motivation is well grounded in real operational concerns for space mission support, and the authors provide public baseline code and a clear accuracy metric. These are genuine strengths. However, the central scientific claim—that malicious LLM modifications produce detectable text-level signals and that the resulting detectors are universal—is unsupported by any empirical evidence in this manuscript. The examples in Appendix B show that some fake texts contain obvious artifacts, but the paper itself concedes in Section 1.5 that the task is trivial for some articles and requires advanced NLP for others. Without quantitative validation, the benchmark's validity as a measure of generalizable detection remains unestablished. The paper is better characterized as a competition announcement or proposal than as a complete research contribution in its current form.

major comments (4)

[Sections 1.4, 1.5, and 1.7] The load-bearing premise that malicious LLM modifications produce detectable text-level signals, and that the task is neither trivially easy nor impossible, is asserted but never tested. Section 1.5 states that 'this task can be trivial for some articles and solved by filtering specific keywords or Unicode characters,' and the Section 1.7 baselines rely on exactly such language- and character-based filters. Yet the paper reports no baseline accuracy, no chance-level comparison, no distribution of pair difficulty, and no human performance. Please add a pilot analysis on the training set: report the accuracy of the provided baselines, the fraction of pairs that are trivially separable by the proposed filters, the distribution of baseline confidence, and an estimate of the ceiling from simple feature-based classifiers. Without this, the abstract's assertion that the competition addresses the two security threats with transferable methods is not supported.
[Section 1.2 and 1.4] The claim that 'the methods, algorithms, and tools proposed in this competition are universal and can be applied to textual data, regardless of its topic' is unsupported by the evidence. All data come from a single journal, The Messenger, with a narrow topical and stylistic range, and the attack definitions are withheld in Section 1.4, so external readers cannot judge how domain-specific the artifacts are. To make the universality claim credible, the paper must either provide cross-domain validation (for example, applying the winning method to out-of-domain text pairs) or substantially temper the claim to a domain-specific benchmark and discuss generalization limits explicitly.
[Section 1.6, Eq. (1)] Equation (1) defines accuracy, but the paper does not state whether the test set is balanced or whether the assignment of labels '1' and '2' to the benign and fake summaries is randomized per pair. If the position of the benign summary is not randomized, or if the test set has a skewed class balance, accuracy can be inflated by a trivial position-majority or label-majority classifier. Please state the randomization procedure and report the majority-class baseline accuracy, as well as accuracy conditioned on the position of the real text if applicable.
[Sections 2.1–2.3] The paper reports participation numbers in Section 2.2 but no results from the leaderboard or from the final evaluation. Since the scientific contribution is the benchmark itself, the paper should include, after the competition ends, the distribution of public and private leaderboard accuracies, the winning solutions' methods and scores, and a comparison with the provided baselines. This is a missing result rather than a conceptual flaw, but it is necessary for the paper to substantiate its central claim.

minor comments (5)

[Abstract and throughout] There are several typographical and grammatical issues, for example 'the proper output from LLM' should be 'from an LLM', and the metric formula in Section 1.6 shows 'nX' instead of a summation symbol and uses '1 {yi = ˆyi}' without defining the indicator notation.
[Section 1.6] The term 'subarticles' is undefined; please clarify whether it refers to individual summaries or to the pairs of summaries.
[Figure 1] The caption 'The graphical summary of the data generation process' is too terse; the figure should label the steps and explicitly define what 'proper' and 'malicious' mean in the diagram.
[Appendix B] The examples are helpful, but there is no annotation of which attack type produced each fake text; adding such labels in a post-competition version would make the appendix more informative.
[References] Some reference entries contain formatting errors, such as a stray space in 'Passi and V orvoreanu' and the misspelling 'Mancinni' for the Ledger and Mancini reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: this is a competition-organization paper with no fitted quantities, no derived predictions, and no load-bearing self-citation chain.

full rationale

The paper describes a Kaggle competition rather than a derivation. Its only equation is the accuracy metric ACC, which is a scoring definition, not a predictive claim obtained from fitted inputs. The central assertion — that the competition addresses data poisoning and overreliance — is presented as motivation, not as a result derived from the data. Section 1.4 explicitly withholds the attack definitions, so no hidden equivalence between the data-generation procedure and the detection task can be exhibited. The statements in Section 1.5 that the task can be trivial for some articles and that baselines in Section 1.7 rely on keyword/character filters raise a feasibility or validity concern about the benchmark, but the reviewing rules for this pass require quoting a specific reduction of a claimed result to its inputs; no such reduction exists here. The self-citations (Kotowski et al., 2025a,b) are contextual references to the authors' related project and earlier competition, and they are not invoked to justify any mathematical step or to forbid alternatives. Therefore the paper is self-contained in the sense relevant to circularity: it makes no fitted-quantity-becomes-prediction move and no ansatz-smuggling move. Score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters and no invented entities. It relies on domain assumptions about the reality of the threats, the detectability of attacks, and the representativeness of the source data. These assumptions are reasonable starting points for a competition but are not empirically supported in the text.

assumptions (3)

domain assumption Data poisoning and overreliance are realistic threats for LLM-based space operations.
Invoked in Section 1.2 to motivate the competition; supported by citations, but not by new evidence in this paper.
domain assumption Maliciously modified LLMs produce detectable text artifacts.
Assumed in Sections 1.4 and 1.5; no validation is provided, and attack details are withheld.
domain assumption Text from The Messenger journal is representative of space operations text.
Used to build the dataset in Section 1.4; no analysis supports representativeness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fake or Real: The Impostor Hunt in Texts for Space Operations." pith.science (2026). https://pith.science/paper/IA7E7XQD

@misc{pith2026250713508,
  author       = {Pith},
  title        = {Pith review of: Fake or Real: The Impostor Hunt in Texts for Space Operations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IA7E7XQD}},
  note         = {Machine review of arXiv:2507.13508}
}

read the original abstract

The "Fake or Real" competition hosted on Kaggle (https://www.kaggle.com/competitions/fake-or-real-the-impostor-hunt ) is the second part of a series of follow-up competitions and hackathons related to the "Assurance for Space Domain AI Applications" project funded by the European Space Agency (https://assurance-ai.space-codev.org/ ). The competition idea is based on two real-life AI security threats identified within the project -- data poisoning and overreliance in Large Language Models. The task is to distinguish between the proper output from LLM and the output generated under malicious modification of the LLM. As this problem was not extensively researched, participants are required to develop new techniques to address this issue or adjust already existing ones to this problem's statement.

Figures

Figures reproduced from arXiv: 2507.13508 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

[1]

Trojan horse hunt in time series forecasting for space operations

Krzysztof Kotowski, Ramez Shendy, Jakub Nalepa, Przemyslaw Biecek, Agata Kaczmarek, Artur Janicki, Evridiki Ntagiou, et al. Trojan horse hunt in time series forecasting for space operations. arXiv preprint arXiv:2506.01849, 2025 a

arXiv 2025
[2]

Towards Explainable and Secure AI for Space Mission Operations

Krzysztof Kotowski, Piotr Wilczynski, Dawid Pludowski, Agata Kaczmarek, Ramez Shendy, Jakub Nalepa, Przemyslaw Biecek, and Evridiki Ntagiou. Towards Explainable and Secure AI for Space Mission Operations . In 2025 SpaceOps Conference , Montreal, Canada, 2025 b . Canadian Space Agency

work page 2025
[3]

Data poisoning attacks against online learning, 2018

Yizhen Wang and Kamalika Chaudhuri. Data poisoning attacks against online learning, 2018

work page 2018
[4]

Targeted backdoor attacks on deep learning systems using data poisoning, 2017

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017

2017
[5]

Data poisoning in llms: Jailbreak-tuning and scaling laws, 2024

Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kellin Pelrine. Data poisoning in llms: Jailbreak-tuning and scaling laws, 2024

work page 2024
[6]

Data poisoning in deep learning: A survey, 2025

Pinlong Zhao, Weiyao Zhu, Pengfei Jiao, Di Gao, and Ou Wu. Data poisoning in deep learning: A survey, 2025

work page 2025
[7]

Cohen, David Krueger, and Fazl Barez

Tingchen Fu, Mrinank Sharma, Philip Torr, Shay B. Cohen, David Krueger, and Fazl Barez. Poisonbench: Assessing large language model vulnerability to data poisoning, 2025

work page 2025
[8]

Medical large language models are vulnerable to data-poisoning attacks

Daniel Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly Valliani, Jeff Zhang, Gabriel Rosenbaum, Ashley Amend-Thomas, David Kurland, Caroline Kremer, Alexander Eremiev, Bruck Negash, Daniel Wiggan, Michelle Nakatsuka, Karl Sangwon, Sean Neifert, Hammad Khan, Akshay Save, and Eric Oermann. Medical large language models are vulnerable to data...

work page 2025

Show all 21 references

[9]

Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift, 2019

2019
[10]

Adversarial concept drift detection under poisoning attacks for robust data stream mining

Lukasz Korycki and Bartosz Krawczyk. Adversarial concept drift detection under poisoning attacks for robust data stream mining. Machine Learning, 112, 2023

2023
[11]

Overreliance on ai: Literature review

Samir Passi and Mihaela Vorvoreanu. Overreliance on ai: Literature review. Microsoft Research, 2022

2022
[12]

Trust and reliance on ai — an experimental study on the extent and costs of overreliance on ai

Artur Klingbeil, Cassandra Gruetzner, and Philipp Schreck. Trust and reliance on ai — an experimental study on the extent and costs of overreliance on ai. Computers in Human Behavior, 160: 0 108352, 2024. ISSN 0747-5632

2024
[13]

Explanatory model analysis

Przemyslaw Biecek and Tomasz Burzykowski. Explanatory model analysis. 2021

2021
[14]

Explanations can reduce overreliance on ai systems during decision-making, 2023

Helena Vasconcelos, Matthew Joerke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. Explanations can reduce overreliance on ai systems during decision-making, 2023

2023
[15]

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco T \' u lio Ribeiro, and Daniel S. Weld. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. CoRR, abs/2006.14779, 2020

2006 arXiv
[16]

Llm-check: Investigating detection of hallucinations in large language models

Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. Llm-check: Investigating detection of hallucinations in large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang...

2024
[17]

Detecting llm hallucinations using monte carlo simulations on token probabilities

Grant Ledger and Rafael Mancinni. Detecting llm hallucinations using monte carlo simulations on token probabilities. June 2024

2024
[18]

Interrogatellm: Zero-resource hallucination detection in llm-generated answers, 2024

Yakir Yehuda, Itzik Malkiel, Oren Barkan, Jonathan Weill, Royi Ronen, and Noam Koenigstein. Interrogatellm: Zero-resource hallucination detection in llm-generated answers, 2024

2024
[19]

Steer llm latents for hallucination detection, 2025

Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, and Yixuan Li. Steer llm latents for hallucination detection, 2025

2025
[20]

A survey of sentiment analysis: Approaches, datasets, and future research

Kian Long Tan, Chin Poo Lee, and Kian Ming Lim. A survey of sentiment analysis: Approaches, datasets, and future research. Applied Sciences, 13 0 (7): 0 4550, 2023

2023
[21]

Part of speech tagging: a systematic review of deep learning and machine learning approaches

Alebachew Chiche and Betselot Yitagesu. Part of speech tagging: a systematic review of deep learning and machine learning approaches. Journal of Big Data, 9 0 (1): 0 10, 2022

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Trojan horse hunt in time series forecasting for space operations

Krzysztof Kotowski, Ramez Shendy, Jakub Nalepa, Przemyslaw Biecek, Agata Kaczmarek, Artur Janicki, Evridiki Ntagiou, et al. Trojan horse hunt in time series forecasting for space operations. arXiv preprint arXiv:2506.01849, 2025 a

arXiv 2025

[2] [2]

Towards Explainable and Secure AI for Space Mission Operations

Krzysztof Kotowski, Piotr Wilczynski, Dawid Pludowski, Agata Kaczmarek, Ramez Shendy, Jakub Nalepa, Przemyslaw Biecek, and Evridiki Ntagiou. Towards Explainable and Secure AI for Space Mission Operations . In 2025 SpaceOps Conference , Montreal, Canada, 2025 b . Canadian Space Agency

work page 2025

[3] [3]

Data poisoning attacks against online learning, 2018

Yizhen Wang and Kamalika Chaudhuri. Data poisoning attacks against online learning, 2018

work page 2018

[4] [4]

Targeted backdoor attacks on deep learning systems using data poisoning, 2017

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017

2017

[5] [5]

Data poisoning in llms: Jailbreak-tuning and scaling laws, 2024

Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kellin Pelrine. Data poisoning in llms: Jailbreak-tuning and scaling laws, 2024

work page 2024

[6] [6]

Data poisoning in deep learning: A survey, 2025

Pinlong Zhao, Weiyao Zhu, Pengfei Jiao, Di Gao, and Ou Wu. Data poisoning in deep learning: A survey, 2025

work page 2025

[7] [7]

Cohen, David Krueger, and Fazl Barez

Tingchen Fu, Mrinank Sharma, Philip Torr, Shay B. Cohen, David Krueger, and Fazl Barez. Poisonbench: Assessing large language model vulnerability to data poisoning, 2025

work page 2025

[8] [8]

Medical large language models are vulnerable to data-poisoning attacks

Daniel Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly Valliani, Jeff Zhang, Gabriel Rosenbaum, Ashley Amend-Thomas, David Kurland, Caroline Kremer, Alexander Eremiev, Bruck Negash, Daniel Wiggan, Michelle Nakatsuka, Karl Sangwon, Sean Neifert, Hammad Khan, Akshay Save, and Eric Oermann. Medical large language models are vulnerable to data...

work page 2025