Pith. sign in

REVIEW 3 major objections 6 minor 75 references

The paper argues that the file format carrying a document is not a neutral wrapper: swapping TXT for CSV can flip a workflow's decision in over 41% of instances and cut accuracy by more than half, so format robustness should be a first-clas

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:03 UTC pith:BDYULGF2

load-bearing objection Read this for the measurement, not the mitigation: the core finding survives, but the semantic-equivalence check and in-sample routing evaluation need work. the 3 major comments →

arxiv 2607.27648 v1 pith:BDYULGF2 submitted 2026-07-30 cs.SE

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

Xiaoyu Zhang , Xianyun Cheng , Tianlin Li , Yuwei Zheng , Yue Yang , Yang Liu This is my paper
classification cs.SE
keywords LLM document workflowsformat robustnessmetamorphic testingmetamorphic relationsLLM testingdecision driftevidence driftexecution stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that end-to-end LLM document workflows are systematically sensitive to the file format of their input. Using a metamorphic testing framework with three relations, it shows that when the same semantic content is delivered as TXT, Markdown, JSON, or CSV, workflows frequently change their final decisions, shift the evidence they cite, and become less stable across repeated runs. Across 48,000 executions of four workflows on four high-stakes tasks, decision-outcome invariance was violated in 41.33% of instances on average, with the largest single-format accuracy drop reaching 56 percentage points. It also shows that a lightweight client-side format-routing step, which converts files to a per-workflow optimal format before inference, recovers up to 44.21% of those decision violations without retraining. If correct, document format must be treated as a reliability factor in LLM software engineering, not just an interface detail.

Core claim

The central discovery is that format-induced behavioral drift is pervasive, directional, and partly correctable. The same medical or financial record rendered in different formats is not treated the same way: structured formats such as CSV are systematically more likely than plain text to trigger wrong decisions, and the worst-performing format depends on the workflow's parsing and serialization path. The paper further finds that evidence-level drift can occur silently—36.44% of evidence-drift cases leave the final decision unchanged—and that execution stability under a format can fall as low as 11%, meaning a workflow that is deterministic under one format can become effectively nondetermin

What carries the argument

The framework's load-bearing objects are three metamorphic relations defined over a format rendering function: MR1 decision outcome invariance, MR2 reasoning evidence invariance, and MR3 execution stability invariance. The principle is that for semantically equivalent renderings of the same instance, the workflow's final decision, its extracted evidence set, and its stability across repeated runs should be unchanged; any violation is counted as a format-induced fault. The testing pipeline converts each source instance into a unified intermediate representation, renders it into TXT, Markdown, JSON, and CSV under constraints of semantic invariance and instruction isolation, runs the variants t

Load-bearing premise

The main load-bearing premise is that the four file renderings really are semantically equivalent, so any difference in workflow behavior is caused by format; the paper's only quantitative check is two experts judging 20 transformation groups, a small sample for a claim of semantic invariance.

What would settle it

Take a sample of the paper's tasks and have multiple independent annotators verify that the TXT, Markdown, JSON, and CSV versions of each instance carry identical facts, field values, and relationships; if a nontrivial fraction of conversions are judged to shift semantics or salience, the reported violation rates conflate format effects with conversion artifacts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Document format should become a standard evaluation variable in LLM software testing, alongside prompts and model versions.
  • Deployers of high-stakes workflows such as medical triage and credit decisions should test for format-induced decision flips before trusting structurally valid outputs.
  • Format routing, converting inputs to a workflow-specific preferred format before inference, is a cheap and training-free mitigation that directly reduces decision, evidence, and stability violations.
  • Aggregate accuracy comparisons across formats can hide large instance-level inconsistency, so safety evaluation needs instance-level metamorphic checks.
  • Structured formats like CSV deserve special scrutiny because they tended to produce the largest accuracy drops and stability failures in this study.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If format-induced evidence drift generalizes, then document-centric systems beyond the four workflows studied, such as retrieval-augmented pipelines or document agents, are likely to inherit the same sensitivity; the paper names this as future work.
  • A testable consequence of the routing result is that optimal-format mappings should be re-calibrated when a model or workflow is updated; static offline routing may silently become stale.
  • The silent-evidence-drift finding suggests audit and interpretability tools for LLM workflows should record input-format provenance; otherwise they can report the same decision with a completely changed reasoning basis.
  • A plausible mechanistic hypothesis for the CSV effect is that exposed field boundaries and headers make demographic attributes more salient to the model; reordering columns or adding semantically weighted headers is a cheap experiment that could test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a format-aware metamorphic testing framework for end-to-end LLM document workflows. It defines three metamorphic relations (MR1 decision-outcome invariance, MR2 reasoning-evidence invariance, MR3 execution-stability invariance) and applies them to four production workflows (OpenAI, Anthropic, Google, Alibaba), four datasets (MedQA, Construct, DiscrimEval, Credit Card), and four document formats (TXT, MD, JSON, CSV), totaling 48,000 executions. The authors report that format variation causes widespread MR1 violations (avg. 41.33%), evidence drift (avg. 25.14%), and stability degradation (avg. 45.33%), with accuracy dropping by up to 56.00%. They further propose two test-time mitigation strategies — vote aggregation and format routing — and claim the routing strategy reduces MR1 violations by up to 44.21% without model retraining. The central claim is that document format is a non-trivial reliability factor in LLM document workflows.

Significance. If the empirical findings are sound, the paper makes a significant contribution to LLM system testing: it identifies a previously underexplored failure mode (format-induced behavioral inconsistency) and provides a reusable metamorphic-testing methodology. The study is large and comparatively well-controlled: four real workflows, four datasets, four formats, repeated executions, and appropriate statistical tests (Cochran's Q, McNemar with Holm correction, bootstrap CIs). The authors also release code and data, which supports reproducibility. The practical implication — that format robustness should be a standard evaluation axis in high-stakes LLM deployments — is timely and actionable. However, two load-bearing issues (semantic equivalence of format variants and evaluation leakage in the routing mitigation) must be addressed before the headline numbers can be taken at face value.

major comments (3)
  1. All three MRs and the headline MRV/accuracy numbers assume that the four renderings φ(x,f) are semantically equivalent, so any behavioral difference is attributable to format. The only quantitative check reported is two experts confirming 100% consistency on 20 sampled transformation groups (§6 threat 2) — 0.5% of the 4,000 groups. The paper itself acknowledges 'Variations in format expressiveness could theoretically introduce semantic shifts' (§6). This validation is too weak for a load-bearing premise: if the conversion operator T(C_x,f) drops or alters fields (e.g., CSV escaping, JSON flattening, TXT omitting a field), MR1/MR2 violations could occur even for a perfectly format-robust workflow. I recommend adding an automated round-trip equivalence check on all instances (parse each render back to C_x and verify field/value/relationship preservation) and/or a much larger stratified hum
  2. The format routing strategy selects the 'empirically optimal format' from offline empirical results and then reports ΔMRV on the same datasets (Construct and MedQA). This is a train-on-test evaluation: the routing table is exactly the per-format empirical performance on the very instances used to compute the 44.21% reduction. The reported improvement is therefore an upper bound and will shrink on new instances. Please evaluate routing with a held-out split (e.g., calibrate on one half of a dataset, test on the other) or with cross-validation, and report the distribution of ΔMRV across folds. This is necessary before claiming 'recover up to 44.21%' as a deployable mitigation.
  3. The voting-strategy ΔM3 values are exactly the original M3 values in Table 1 (e.g., MedQA OpenAI 71.37, Construct OpenAI 86.40). After majority voting over 3 runs, the system produces a single decision, so the MR3 stability metric — which requires comparing decisions across k repeated executions — is trivially satisfied. Reporting these numbers as evidence that voting improves execution stability is a definitional artifact. The comparison between voting and routing on M3 is therefore not meaningful as presented. Please either drop the M3 rows for voting or define a repeated-voting ensemble experiment where the ensemble is executed multiple times.
minor comments (6)
  1. [Abstract vs §5.2] The abstract reports 'accuracy to drop by up to 53.63%', while §5.2 and Table 2 report a maximum gap of 56.00% (Construct, Alibaba CSV). Please reconcile or clarify the metric used for the abstract value.
  2. [§5.4] The text states 'we collect and analyze 3,000 groups (k=3) of repeated execution results.' Given the setup (§5.1: 4 workflows × 4 datasets × 250 instances), the instance count is 4,000; if 'group' refers to an instance-format combination, the number should be 16,000. Please clarify the count.
  3. [References] Reference [4] is listed as 'Authors. 2026. Open-sourced Data.' In a non-anonymous version this should be replaced with the actual author team or a proper repository citation.
  4. [§5.1 (MR2 threshold)] The MR2 threshold θ=0.3 is validated on 20 expert-checked samples (§5.1). A sensitivity analysis over θ (e.g., 0.1–0.7) would help establish that the MR2 violation rates and the '36.44% silent violations' claim are not threshold artifacts.
  5. [§5.2] The set of '3,302 instances' used for the error-format analysis is not defined before the result is quoted. Please state explicitly how this subset is constructed (e.g., instances with at least one correct and one incorrect format decision across all workflows/datasets).
  6. [§5.1 (Construct dataset)] The Construct dataset is self-constructed (§5.1). Please include a summary table with dataset size, number of fields, and label balance, so readers can assess how representative the multi-field scenarios are.

Circularity Check

1 steps flagged

RQ4's format-routing gain is fitted on the same data used to score it; the core RQ1–3 findings remain external measurements.

specific steps
  1. fitted input called prediction [Section 5.5 (RQ4: Mitigation Strategies), 'Format Routing' paragraph and Table 3]
    "It inserts a lightweight routing layer upstream of the workflow that converts every input document to the empirically optimal format before inference. Concretely, this layer maps the input task domain to the optimal format for the current workflow based on offline empirical results. For example, a CSV MedQA input to the OpenAI workflow is automatically converted to MD (its optimal format) prior to inference. Leveraging the pre-collected empirical results rather than per-document LLM analysis, this approach introduces negligible overhead."

    The router is configured by picking the empirically best format from the same pre-collected executions on the same datasets/workflows that Table 3 then uses to compute the claimed 44.21% average reduction in M1. Because the optimal format is selected to perform best on these exact instances, the reported 'reduction' is the in-sample best-format-versus-mixed-format gap, not an independent evaluation of a routing policy. No held-out split or cross-validation is described for the routing experiment, so the RQ4 headline mitigation number is statistically forced by the selection step.

full rationale

The main empirical contribution (RQ1–RQ3) is not circular: MR1/MR2/MR3 violation rates and per-format accuracies are measured against ground-truth labels and repeated workflow outputs, providing external anchors. The paper's acknowledged semantic-equivalence assumption (Threat 2) is a validity threat that could confound interpretation, but it is not a reduction by construction. The clear circular step is the format-routing evaluation in RQ4: the empirically optimal format is chosen from the same offline results on which the 44.21% MR1 reduction is later computed, so that mitigation claim is in-sample fitting rather than independent prediction. The central finding that format variation correlates with decision flips and accuracy drops does not depend on this fitted value; for that reason the overall score is partial (6) rather than fully circular.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central empirical finding is a measurement, not a derivation, so the ledger is small. The main freely chosen quantities are the MR2 threshold θ and the format-routing optimal-format mapping; both are calibrated on the same data used for evaluation, which is the largest circularity burden. No new physical or mathematical entities are introduced.

free parameters (2)
  • MR2 evidence-similarity threshold θ = 0.3
    Threshold below which evidence sets count as drifted; chosen on 20 expert-labeled samples as minimizing false positives/negatives (Section 5.1), so calibrated on the same evaluation material.
  • Optimal format per workflow-task in format routing = e.g., MD for OpenAI on MedQA; varies per provider/dataset
    Section 5.5: the routing layer maps each task domain to the 'empirically optimal format' from pre-collected results; these choices are fitted to the same datasets later used to compute the reported ΔMRV gains.
axioms (5)
  • domain assumption Semantically equivalent inputs should yield equivalent workflow outputs (format robustness principle).
    Basis of MR1–MR3 (Sections 2.2 and 4.1); this is the property under test, not a derived theorem.
  • domain assumption The four formats TXT, MD, JSON, CSV can express the same semantic content without loss.
    Section 4.2 transformation constraints and Section 6 threat 2; checked on only 20 transformation groups.
  • domain assumption LLM behavior at temperature 0 with k=3 repeats is sufficient to estimate stability and MR3.
    Section 5.1 and Section 6 threat 4; authors acknowledge k=3 limitations and provide only partial k=5 validation on two datasets.
  • standard math Standard statistical tests (Cochran's Q, McNemar with Holm, Spearman, bootstrap) apply to the comparison of workflow outputs.
    Used in Sections 5.2–5.4; standard assumptions of paired categorical data and resampling.
  • domain assumption Keyword-based mapping of extracted evidence to semantic categories with >90% success is a valid operationalization of reasoning evidence.
    Section 5.1 implementation; unmapped items get similarity 0 as a conservative penalty; validation only on 20 samples for threshold choice.

pith-pipeline@v1.3.0-daily-deepseek · 29581 in / 14450 out tokens · 135150 ms · 2026-08-01T04:03:07.428323+00:00 · methodology

0 comments
read the original abstract

LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users' perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.

Figures

Figures reproduced from arXiv: 2607.27648 by Tianlin Li, Xianyun Cheng, Xiaoyu Zhang, Yang Liu, Yue Yang, Yuwei Zheng.

Figure 2
Figure 2. Figure 2: Overview of the Testing Framework. 4 Design To evaluate the behaviors and reveal potential risks of end-to-end LLM document workflows when confronted with various input formats, we design and implement a format-aware metamorphic testing framework. Different from existing testing methods that treat the LLM as the test object [27, 59], our framework consid￾ers the end-to-end LLM document workflow. The core p… view at source ↗
Figure 3
Figure 3. Figure 3: Erroneous Results Across Formats. • Analysis of MR1 Violation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy Across Formats and Workflows on MedQA and Con￾struct Dataset. More importantly, the degradation pattern is not determined by the format but reflects a significant format-workflow interaction. The worst-performing format varies across workflows. On the Con￾struct dataset, the OpenAI workflow achieves its best accuracy with MD yet its worst with CSV, while the Alibaba workflow exhibits the most seve… view at source ↗
Figure 5
Figure 5. Figure 5: Format Disparity Score Heatmap on DiscrimEval Dataset. TXT JSON MD CSV (a) OpenAI 0 50 100 Stability (%) TXT JSON MD CSV (b) Anthropic 0 50 100 Stability (%) TXT JSON MD CSV (c) Google 0 50 100 Stability (%) TXT JSON MD CSV (d) Alibaba 0 50 100 Stability (%) MedQA Construct DiscrimEval Credit Card [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Stability (𝑆𝑡𝑏) Across Different Datasets and Workflows. direct threats to the interpretability and trustworthiness of LLM document workflows. • Analysis of Evidence Drift. The results in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 3 canonical work pages

  1. [1]

    Open Life Science AI. 2024. medmcqa. https://huggingface.co/datasets/openlife scienceai/medmcqa/tree/main/data Accessed: 2026-03-21

    2024

  2. [2]

    Aliyun. 2026. Long context(Qwen-Long). https://www.alibabacloud.com/help/ en/model-studio/long-context-qwen-long Accessed: 2026-03-21

    2026

  3. [3]

    Anthropic. 2026. Advancing Claude in healthcare and the life sciences. https: //www.anthropic.com/news/healthcare-life-sciences. Accessed: 2026-03-21

    2026

  4. [4]

    Authors. 2026. Open-sourced Data. https://doi.org/10.5281/zenodo.21696268 Accessed: 2026-07-30

    work page doi:10.5281/zenodo.21696268 2026

  5. [5]

    Israel Abebe Azime, Deborah D Kanubala, Tejumade Afonja, Mario Fritz, Isabel Valera, Dietrich Klakow, and Philipp Slusallek. 2025. Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serial- ization Approaches. InFindings of the Association for Computational Linguistics: EMNLP 2025. 17478–17503

    2025

  6. [6]

    Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. InProceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 194–199

    2024

  7. [7]

    Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May...

    2025

  8. [8]

    Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases.arXiv preprint arXiv:2002.12543 (2020)

    Pith/arXiv arXiv 2020

  9. [9]

    Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities.ACM Computing Surveys (CSUR)51, 1 (2018), 1–27

    2018

  10. [10]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning

    2024

  11. [11]

    Steven Cho, Stefano Ruberto, and Valerio Terragni. 2025. Metamorphic testing of large language models for natural language processing. In2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 174–186

    2025

  12. [12]

    Xu Chu, Zhijie Tan, Hanlin Xue, Guanyu Wang, Tong Mo, and Weiping Li

  13. [13]

    Claude. 2025. ClaudeAPIDocs, Files API. https://platform.claude.com/docs/en/ build-with-claude/files Accessed: 2026-03-21

    2025

  14. [14]

    W. G. COCHRAN. 1950. THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES.Biometrika37, 3-4 (12 1950), 256–266. arXiv:https://academic.oup.com/biomet/article-pdf/37/3-4/256/421327/37- 3-4-256.pdf doi: 10.1093/biomet/37.3-4.256

    work page doi:10.1093/biomet/37.3-4.256 1950

  15. [15]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-Verification Reduces Hallucina- tion in Large Language Models. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL, Vol. ACL 2024), Lun-We...

    doi:10.18653/v1/2024 2024

  16. [16]

    Alastair F Donaldson and Andrei Lascu. 2016. Metamorphic testing for (graphics) compilers. InProceedings of the 1st international workshop on metamorphic testing. 44–47

    2016

  17. [17]

    Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, and Dongmei Zhang. 2024. Encoding Spreadsheets for Large Language Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computatio...

    work page doi:10.18653/v1/2024.emnlp-main.1154 2024

  18. [18]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engi- neering: Survey and Open Problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). 31–53. doi: 10.1109/ICSE-FoSE59343.2023.00008

    arXiv 2023

  19. [19]

    Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. 2025. Automatically Bench- marking LLM Code Agents through Agent-Driven Annotation and Evaluation. CoRRabs/2510.24358 (2025). arXiv:2510.24358 doi: 10.48550/ARXIV.2510.24358

    doi:10.48550/arxiv.2510.24358 2025

  20. [20]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and Fairness in Large Language Models: A Survey.Computational Linguistics50, 3 (Sept. 2024), 1097–1179. doi: 10.1162/coli_a_00524

    doi:10.1162/coli_a_00524 2024

  21. [21]

    Gemini. 2026. Gemini API, Document understanding. https://ai.google.dev/ge mini-api/docs/document-processing Accessed: 2026-03-21

    2026

  22. [22]

    Google. 2026. Gemini API, Files API. https://ai.google.dev/gemini-api/docs/files Accessed: 2026-03-21

    2026

  23. [23]

    Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. InProceedings of the 30th International Conference on Neural Information Processing Systems(Barcelona, Spain)(NIPS’16). Curran Associates Inc., Red Hook, NY, USA, 3323–3331

    2016

  24. [24]

    Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre- training. InProceedings of the 58th annual meeting of the association for computa- tional linguistics. 4320–4333

    2020

  25. [25]

    Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics6, 2 (1979), 65–70. http://www.jstor.org/stable /4615733

    1979

  26. [26]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review.ACM Trans. Softw. Eng. Methodol.33, 8, Article 220 (Dec. 2024), 79 pages. doi: 10.1145/3695988

    doi:10.1145/3695988 2024

  27. [27]

    Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui

    Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias testing and mitigation in llm-based code generation.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–31

    2025

  28. [28]

    Yifan Huang, Xiaojun Jia, Wenbo Guo, Yuqiang Sun, Yihao Huang, Chong Wang, and Yang Liu. 2026. Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking. InProceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2026)

    2026

  29. [29]

    Ali Babar

    Sangwon Hyun, Mingyu Guo, and M. Ali Babar. 2024. METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities. In2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 117–128. doi: 10.1109/ICST60714.2024.00019

    arXiv 2024

  30. [30]

    Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, and Guang- sheng Yu. 2026. SoK: Agentic Skills–Beyond Tool Use in LLM Agents.arXiv preprint arXiv:2602.20867(2026)

    Pith/arXiv arXiv 2026

  31. [31]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences11, 14 (2021), 6421

    2021

  32. [32]

    Venkata Sai Nageen Kanikanti. 2024. LLMs for Financial Document Processing. International Journal of Artificial Intelligence, Data Science, and Machine Learning 5, 1 (2024), 167–172

    2024

  33. [33]

    Wenjun Ke, Yifan Zheng, Yining Li, Hengyuan Xu, Dong Nie, Peng Wang, and Yao He. 2025. Large language models in document intelligence: A comprehensive survey, recent advances, challenges, and future trends.ACM Transactions on Information Systems44, 1 (2025), 1–64

    2025

  34. [34]

    Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matarić, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health.JMIR Mental Health11, 1 (2024), e59479. Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows ASE ’26, October 12–16, 2026, Munich, Germany

    2024

  35. [35]

    Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. 2022. MarkupLM: Pre-training of text and markup language for visually rich document understanding. InProceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6078–6087

    2022

  36. [36]

    Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, and Junbo Zhao. 2025. LongTableBench: bench- marking long-context table reasoning across real-world formats and domains. Findings of the Association for Computational Linguistics: EMNLP2025 (2025)

    2025

  37. [37]

    Penghui Li, Yinxi Liu, and Wei Meng. 2021. Understanding and detecting per- formance bugs in markdown compilers. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 892–904

    2021

  38. [38]

    LLMBias. 2025. BiasLens. https://github.com/LLMBias/BiasLens/tree/main Accessed: 2026-03-21

    2025

  39. [39]

    Quinn McNemar. 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages.Psychometrika12, 2 (1947), 153–157. doi: 10.1007/BF02295996

    doi:10.1007/bf02295996 1947

  40. [40]

    Jonathan Chen Meenu Bondili. 2025. Use Gemini 2.0 to speed up document extraction and lower costs. https://cloud.google.com/blog/products/ai-machine- learning/use-gemini-2-0-to-speed-up-data-processing. Accessed: 2026-03-21

    2025

  41. [41]

    Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, et al . 2024. Exploring the impact of table-to-text methods on augmenting llm-based question answering with domain hybrid data. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2024

  42. [42]

    1993.Bootstrapping: A nonparametric approach to statistical inference

    Christopher Z Mooney, Robert D Duval, and Robert Duvall. 1993.Bootstrapping: A nonparametric approach to statistical inference. Number 95. sage

    1993

  43. [43]

    Păsăreanu

    Shirin Nilizadeh, Yannic Noller, and Corina S. Păsăreanu. 2019. DifFuzz: differ- ential fuzzing for side-channel analysis. InProceedings of the 41st International Conference on Software Engineering. 176–187. doi: 10.1109/ICSE.2019.00034

    arXiv 2019

  44. [44]

    Yannic Noller and Saeid Tizpaz-Niari. 2021. QFuzz: quantitative fuzzing for side channels. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 257–269. doi: 10.1145/3460319.3464817

    arXiv 2021

  45. [45]

    OpenAI. 2026. ChatGPT Release Notes: January 7, 2026, Health in ChatGPT. https://help.openai.com/en/articles/6825453-chatgpt-release-notes. Accessed: 2026-03-21

    arXiv 2026

  46. [46]

    OpenAI. 2026. OpenAI API, Code Interpreter. https://developers.openai.com/ap i/docs/guides/tools-code-interpreter Accessed: 2026-03-21

    2026

  47. [47]

    OpenAI. 2026. OpenAI API, File inputs. https://developers.openai.com/api/docs /guides/file-inputs Accessed: 2026-03-21

    2026

  48. [48]

    Stephen R Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, et al. 2024. A toolbox for surfacing health equity harms and biases in large language models.Nature Medicine30, 12 (2024), 3590–3600

    2024

  49. [49]

    Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, Liam G

    Stephen R. Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, Liam G. McCoy, Leo Anthony Celi, Yun Liu, Mike Schaekermann, Alanna Walton, Alicia Parrish, Chirag Nagpal, Preeti Singh, Akeiylah Dewitt, Philip Mansfield, Sushant Prakash, Katherine Heller, Alan...

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.12025 2024

  50. [50]

    Jiarui Qin, Weinan Zhang, Rong Su, Zhirong Liu, Weiwen Liu, Ruiming Tang, Xiuqiang He, and Yong Yu. 2021. Retrieval & Interaction Machine for Tabular Data Prediction. InKDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, Feida Zhu, Beng Chin Ooi, and Chunyan Miao (Eds.). ACM, 1379–1...

    arXiv 2021

  51. [51]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1073–1083. doi: 10.18653/v1/ P17-1099

    doi:10.18653/v1/ 2017

  52. [52]

    Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing.IEEE Transactions on software engineering42, 9 (2016), 805–824

    2016

  53. [53]

    sigpwned. 2023. Popular names by country dataset: A dataset of popular fore- names and surnames by country. https://github.com/sigpwned/popular-names- by-country-dataset Accessed: 2026-03-21

    2023

  54. [54]

    Kuldeep Singh, Simerjot Kaur, and Charese Smiley. 2024. Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline. In Proceedings of the 5th ACM International Conference on AI in Finance. 266–273

    2024

  55. [55]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al . 2025. Toward expert-level medical question answering with large language models. Nature medicine31, 3 (2025), 943–950

    2025

  56. [56]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. InProceedings of the 17th ACM Inter- national Conference on Web Search and Data Mining(Merida, Mexico)(WSDM ’24). Association for Computing Machinery, New York, NY, USA, 645–65...

    arXiv 2024

  57. [58]

    Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions.arXiv preprint arXiv:2312.03689(2023)

    Pith/arXiv arXiv 2023

  58. [59]

    Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R Lyu. 2023. Biasasker: Measuring the bias in conversational ai system. InPro- ceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 515–527

    2023

  59. [60]

    Wanderer. 2024. data jobs by lukebarousse. https://www.kaggle.com/datasets/ wandererfakeer/data-jobs-by-lukebarousse Accessed: 2026-03-21

    2024

  60. [61]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision.IEEE Trans. Softw. Eng.50, 4 (April 2024), 911–936. doi: 10.1109/TSE. 2024.3368208

    arXiv 2024

  61. [62]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (2024), 186345

    2024

  62. [63]

    Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2025. Large Language Model Supply Chain: A Research Agenda.ACM Trans. Softw. Eng. Methodol.34, 5, Article 147 (May 2025), 46 pages. doi: 10.1145/3708531

    doi:10.1145/3708531 2025

  63. [64]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=1P L1NIMMrw

    2023

  64. [65]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents.Proc. ACM Softw. Eng.2, FSE, Article FSE037 (June 2025), 24 pages. doi: 10.1145/3715754

    doi:10.1145/3715754 2025

  65. [66]

    Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing.Journal of Systems and Software84, 4 (2011), 544–558

    2011

  66. [67]

    I-Cheng Yeh and Che hui Lien. 2009. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients.Expert Systems with Applications36, 2, Part 1 (2009), 2473–2480. doi: 10.1016/j.eswa .2007.12.020

    doi:10.1016/j.eswa 2009

  67. [68]

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2025. Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1(Toronto ON, Canada)(KDD ’25). Association for Computing Machinery, New York,...

    arXiv 2025

  68. [69]

    Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. InPro- ceedings of the 58th annual meeting of the association for computational linguistics. 8413–8426

    2020

  69. [70]

    Zeeshanahmad4. 2024. Zip Code Data for All Countries and Cities in the World. https://github.com/Zeeshanahmad4/Zip-code-of-all-countries-cities-in-the-wo rld-CSV-TXT-SQL-DATABASE Accessed: 2026-03-21

    2024

  70. [71]

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. InFindings of the Association for Computational Linguistics: ACL 2024. 10471–10506

    2024

  71. [72]

    Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons.IEEE Transactions on Software Engineering48, 1 (2020), 1–36

    2020

  72. [73]

    Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. 2025. Jailguard: A universal detection framework for prompt-based attacks on llm systems.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–40

    2025

  73. [74]

    Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2015. Metamorphic test- ing for software quality assessment: A study of search engines.IEEE Transactions on Software Engineering42, 3 (2015), 264–284

    2015

  74. [75]

    Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholam- reza Haffari, and Yuan-Fang Li. 2023. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 1090–1102

    2023

  75. [2025]

    InFindings of the Association for Computational Linguistics: ACL 2025

    Domaino1s: Guiding llm reasoning for explainable answers in high-stakes domains. InFindings of the Association for Computational Linguistics: ACL 2025. 3275–3293

    2025