{"id":"c270b5ea-0930-4ab2-8ea4-29ecb9dfd38e","arxiv_id":"2607.27648","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Changing the file format of identical input content changes LLM workflow decisions in 41% of cases on average and can reduce accuracy by up to 56 percentage points, with CSV the most error-prone format.","lead":"This paper tests whether LLM document workflows—systems that read uploaded files and produce decisions—still give the same answer when the same content is provided as TXT, Markdown, JSON, or CSV. Across four commercial workflows and four datasets, switching formats changed decisions in about 41% of cases and cut accuracy by up to 56 percentage points; a simple format-routing wrapper recovered much of the loss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic equivalence of format variants rests on a 20-sample manual check; unverified conversion errors could inflate the reported MRV and accuracy-drop figures.","rationale":"The paper is a well-executed empirical study with a transparent methodology and a clear threat discussion. It makes a strong claim that document format is not neutral, supported by three complementary MRs and paired statistical tests. The single most load-bearing assumption is that the four format variants produced by §4.2 are semantically equivalent; the paper explicitly acknowledges this as a threat (§6). The control—20 manually checked transformation groups—is too small to guarantee fidelity across all 4,000 variants, particularly for structured formats like CSV where data loss or reordering is plausible. If even a small fraction of conversions are unfaithful, the MRV numbers overstate format-induced brittleness because the workflow is being handed different facts, not just different wrappers. This concern is concrete and testable without re-running the whole study, and it aligns with the reader's identified weakest assumption. Other issues—θ calibration on 20 samples, k=3 repeats, and in-sample routing gains—affect secondary numbers or the mitigation claim, but the semantic-equivalence premise underwrites the central threat claim. The existing CONDITIONAL verdict remains appropriate: the paper should not be treated as fully accepted until the released artifact provides strong evidence that the conversion pipeline preserves all facts. My proposed round-trip fidelity test on 100 groups would supply that evidence or expose a conversion artifact, without requiring massive compute.","tokens_in":30103,"tokens_out":5428,"duration_ms":57235,"concrete_test":"Using the released pipeline, randomly sample 100 transformation groups (25 per dataset) and run a deterministic round-trip fidelity check: parse each of the four format files back into the intermediate representation C_x via format-aware parsers, then compare field-by-field (and relationship-by-relationship) against the original C_x. Exclude any group where a value is missing, altered, or reordered and recompute Table 1's M1 and M2 averages on the remaining groups. If the averages drop by more than 5 percentage points (e.g., M1 from 41.33% to below 36%), the conversion pipeline is introducing semantic artifacts and the headline severity figures must be revised; if they stay within 2 points, the 20-sample check was sufficient and the original claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that format variation causes up to 53.63% accuracy drops and 41.33% MR1 violations—depends on the premise that the four renderings produced in §4.2 are semantically equivalent, so any behavioral difference is 'solely attributed to the workflow’s response to format changes' (§4.2, constraint 3). The transformation operator T(C_x, f) is the only bridge between source instance and format variants; if T omits or alters a field in CSV/JSON (e.g., a nested JSON value flattened, a CSV cell corrupted by a comma, a TXT narrative dropping a number), the workflow receives genuinely different inputs and MR1/MR2 violations would occur even for a perfectly format-robust system. The authors' only check for this is 20 manually inspected transformation groups (§6, threat 2)—0.5% of the 4,000 groups (4 formats × 1,000 instances). This sample is far too small to detect systematic per-dataset conversion failures, especially in the Construct dataset with its many fields. The paper explicitly acknowledges the risk: 'Variations in format expressiveness could theoretically introduce semantic shifts' (§6, threat 2). Because all three MRs, Table 1's MRV numbers, and the accuracy-gap findings rest on this equivalence, it is the load-bearing assumption most likely to change the headline conclusions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a format-aware metamorphic testing framework for end-to-end LLM document workflows. It defines three metamorphic relations (MR1 decision-outcome invariance, MR2 reasoning-evidence invariance, MR3 execution-stability invariance) and applies them to four production workflows (OpenAI, Anthropic, Google, Alibaba), four datasets (MedQA, Construct, DiscrimEval, Credit Card), and four document formats (TXT, MD, JSON, CSV), totaling 48,000 executions. The authors report that format variation causes widespread MR1 violations (avg. 41.33%), evidence drift (avg. 25.14%), and stability degradation (avg. 45.33%), with accuracy dropping by up to 56.00%. They further propose two test-time mitigation strategies — vote aggregation and format routing — and claim the routing strategy reduces MR1 violations by up to 44.21% without model retraining. The central claim is that document format is a non-trivial reliability factor in LLM document workflows.","tokens_in":30278,"tokens_out":8259,"duration_ms":69584,"significance":"If the empirical findings are sound, the paper makes a significant contribution to LLM system testing: it identifies a previously underexplored failure mode (format-induced behavioral inconsistency) and provides a reusable metamorphic-testing methodology. The study is large and comparatively well-controlled: four real workflows, four datasets, four formats, repeated executions, and appropriate statistical tests (Cochran's Q, McNemar with Holm correction, bootstrap CIs). The authors also release code and data, which supports reproducibility. The practical implication — that format robustness should be a standard evaluation axis in high-stakes LLM deployments — is timely and actionable. However, two load-bearing issues (semantic equivalence of format variants and evaluation leakage in the routing mitigation) must be addressed before the headline numbers can be taken at face value.","major_comments":[{"comment":"All three MRs and the headline MRV/accuracy numbers assume that the four renderings φ(x,f) are semantically equivalent, so any behavioral difference is attributable to format. The only quantitative check reported is two experts confirming 100% consistency on 20 sampled transformation groups (§6 threat 2) — 0.5% of the 4,000 groups. The paper itself acknowledges 'Variations in format expressiveness could theoretically introduce semantic shifts' (§6). This validation is too weak for a load-bearing premise: if the conversion operator T(C_x,f) drops or alters fields (e.g., CSV escaping, JSON flattening, TXT omitting a field), MR1/MR2 violations could occur even for a perfectly format-robust workflow. I recommend adding an automated round-trip equivalence check on all instances (parse each render back to C_x and verify field/value/relationship preservation) and/or a much larger stratified hum","section":null},{"comment":"The format routing strategy selects the 'empirically optimal format' from offline empirical results and then reports ΔMRV on the same datasets (Construct and MedQA). This is a train-on-test evaluation: the routing table is exactly the per-format empirical performance on the very instances used to compute the 44.21% reduction. The reported improvement is therefore an upper bound and will shrink on new instances. Please evaluate routing with a held-out split (e.g., calibrate on one half of a dataset, test on the other) or with cross-validation, and report the distribution of ΔMRV across folds. This is necessary before claiming 'recover up to 44.21%' as a deployable mitigation.","section":null},{"comment":"The voting-strategy ΔM3 values are exactly the original M3 values in Table 1 (e.g., MedQA OpenAI 71.37, Construct OpenAI 86.40). After majority voting over 3 runs, the system produces a single decision, so the MR3 stability metric — which requires comparing decisions across k repeated executions — is trivially satisfied. Reporting these numbers as evidence that voting improves execution stability is a definitional artifact. The comparison between voting and routing on M3 is therefore not meaningful as presented. Please either drop the M3 rows for voting or define a repeated-voting ensemble experiment where the ensemble is executed multiple times.","section":null}],"minor_comments":[{"comment":"The abstract reports 'accuracy to drop by up to 53.63%', while §5.2 and Table 2 report a maximum gap of 56.00% (Construct, Alibaba CSV). Please reconcile or clarify the metric used for the abstract value.","section":"Abstract vs §5.2"},{"comment":"The text states 'we collect and analyze 3,000 groups (k=3) of repeated execution results.' Given the setup (§5.1: 4 workflows × 4 datasets × 250 instances), the instance count is 4,000; if 'group' refers to an instance-format combination, the number should be 16,000. Please clarify the count.","section":"§5.4"},{"comment":"Reference [4] is listed as 'Authors. 2026. Open-sourced Data.' In a non-anonymous version this should be replaced with the actual author team or a proper repository citation.","section":"References"},{"comment":"The MR2 threshold θ=0.3 is validated on 20 expert-checked samples (§5.1). A sensitivity analysis over θ (e.g., 0.1–0.7) would help establish that the MR2 violation rates and the '36.44% silent violations' claim are not threshold artifacts.","section":"§5.1 (MR2 threshold)"},{"comment":"The set of '3,302 instances' used for the error-format analysis is not defined before the result is quoted. Please state explicitly how this subset is constructed (e.g., instances with at least one correct and one incorrect format decision across all workflows/datasets).","section":"§5.2"},{"comment":"The Construct dataset is self-constructed (§5.1). Please include a summary table with dataset size, number of fields, and label balance, so readers can assess how representative the multi-field scenarios are.","section":"§5.1 (Construct dataset)"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a strong contribution once the semantic-equivalence validation and the routing evaluation leakage are fixed. The voting M3 artifact is a relatively simple correction but should be addressed in the revision. The authors should also decide whether the abstract's 53.63% or the body's 56.00% is the headline number."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the measurement, not the mitigation. The paper makes a solid case that document format is not a neutral wrapper in LLM document workflows: across four workflows and four datasets, format changes flip decisions in a large fraction of instances, and the effect is concentrated in structured formats like CSV. The core finding is supported by controlled experiments and appropriate paired statistics. The framework—three metamorphic relations covering decision, evidence, and stability—is a genuinely useful contribution, and it goes beyond prior work that looked at table serialization in isolated tasks. The paper is also honest about its threats and ships code and data.\n\nThe main soft spot is the semantic equivalence assumption. All three MRs assume the four renderings of each instance are semantically identical, so any behavior difference counts as format-induced. The authors verify this with only 20 manually inspected transformation groups (0.5% of the 4,000). The stress-test worry is real, though not fatal: because the variants are generated from a unified intermediate representation via templated transformations rather than free paraphrasing, the risk of systematic info loss is lower than the raw numbers suggest. Still, a few hundred automated round-trip checks would substantially harden the claim.\n\nThe second soft spot is the format-routing mitigation. The optimal format per workflow is selected from the same datasets on which the 44.21% reduction is then computed, so that number is in-sample. It is a decent proof of concept, but the headline reduction is likely optimistic; held-out evaluation would fix it.\n\nMinor issues: the MR2 threshold is calibrated on 20 expert samples (the sensitivity analysis helps), MR3 uses k=3 repeats (the supplementary k=5 ranking check is reassuring), and the abstract's 53.63% accuracy drop conflicts with the 56.00% reported in Section 5.2. Bootstrap CIs should appear in the main tables.\n\nNone of this overturns the central claim. Format variation is a real and under-tested reliability variable, and the paper gives the community a reusable framework for measuring it. Worth a serious referee; expect major but addressable revisions.","headline":"Read this for the measurement, not the mitigation: the core finding survives, but the semantic-equivalence check and in-sample routing evaluation need work.","tokens_in":30905,"tokens_out":2562,"would_cite":true,"duration_ms":26554,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the file format carrying a document is not a neutral wrapper: swapping TXT for CSV can flip a workflow's decision in over 41% of instances and cut accuracy by more than half, so format robustness should be a first-clas","keywords":["LLM document workflows","format robustness","metamorphic testing","metamorphic relations","LLM testing","decision drift","evidence drift","execution stability"],"falsifier":"Take a sample of the paper's tasks and have multiple independent annotators verify that the TXT, Markdown, JSON, and CSV versions of each instance carry identical facts, field values, and relationships; if a nontrivial fraction of conversions are judged to shift semantics or salience, the reported violation rates conflate format effects with conversion artifacts.","tokens_in":29857,"feed_emoji":"📄","tokens_out":5706,"duration_ms":51847,"temperature":0.7,"pith_summary":"The paper tries to establish that end-to-end LLM document workflows are systematically sensitive to the file format of their input. Using a metamorphic testing framework with three relations, it shows that when the same semantic content is delivered as TXT, Markdown, JSON, or CSV, workflows frequently change their final decisions, shift the evidence they cite, and become less stable across repeated runs. Across 48,000 executions of four workflows on four high-stakes tasks, decision-outcome invariance was violated in 41.33% of instances on average, with the largest single-format accuracy drop reaching 56 percentage points. It also shows that a lightweight client-side format-routing step, which converts files to a per-workflow optimal format before inference, recovers up to 44.21% of those decision violations without retraining. If correct, document format must be treated as a reliability factor in LLM software engineering, not just an interface detail.","feed_headline":"Format swap flips LLM workflow decisions 41% of the time","feed_subtitle":"Accuracy can drop by half and decisions can flip; a simple format fix recovers much of the loss.","key_machinery":"The framework's load-bearing objects are three metamorphic relations defined over a format rendering function: MR1 decision outcome invariance, MR2 reasoning evidence invariance, and MR3 execution stability invariance. The principle is that for semantically equivalent renderings of the same instance, the workflow's final decision, its extracted evidence set, and its stability across repeated runs should be unchanged; any violation is counted as a format-induced fault. The testing pipeline converts each source instance into a unified intermediate representation, renders it into TXT, Markdown, JSON, and CSV under constraints of semantic invariance and instruction isolation, runs the variants t","core_discovery":"The central discovery is that format-induced behavioral drift is pervasive, directional, and partly correctable. The same medical or financial record rendered in different formats is not treated the same way: structured formats such as CSV are systematically more likely than plain text to trigger wrong decisions, and the worst-performing format depends on the workflow's parsing and serialization path. The paper further finds that evidence-level drift can occur silently—36.44% of evidence-drift cases leave the final decision unchanged—and that execution stability under a format can fall as low as 11%, meaning a workflow that is deterministic under one format can become effectively nondetermin","pith_inferences":["If format-induced evidence drift generalizes, then document-centric systems beyond the four workflows studied, such as retrieval-augmented pipelines or document agents, are likely to inherit the same sensitivity; the paper names this as future work.","A testable consequence of the routing result is that optimal-format mappings should be re-calibrated when a model or workflow is updated; static offline routing may silently become stale.","The silent-evidence-drift finding suggests audit and interpretability tools for LLM workflows should record input-format provenance; otherwise they can report the same decision with a completely changed reasoning basis.","A plausible mechanistic hypothesis for the CSV effect is that exposed field boundaries and headers make demographic attributes more salient to the model; reordering columns or adding semantically weighted headers is a cheap experiment that could test this."],"forward_implications":["Document format should become a standard evaluation variable in LLM software testing, alongside prompts and model versions.","Deployers of high-stakes workflows such as medical triage and credit decisions should test for format-induced decision flips before trusting structurally valid outputs.","Format routing, converting inputs to a workflow-specific preferred format before inference, is a cheap and training-free mitigation that directly reduces decision, evidence, and stability violations.","Aggregate accuracy comparisons across formats can hide large instance-level inconsistency, so safety evaluation needs instance-level metamorphic checks.","Structured formats like CSV deserve special scrutiny because they tended to produce the largest accuracy drops and stability failures in this study."],"fun_headline_variants":["Format swap flips LLM decisions 41% of the time","CSV vs text: LLM accuracy drops 53% on format change","Format switch breaks LLM workflows, fix recovers 44%","Same content, new format: LLM decisions drift 41%","LLM reliability hinges on document format, study finds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The main load-bearing premise is that the four file renderings really are semantically equivalent, so any difference in workflow behavior is caused by format; the paper's only quantitative check is two experts judging 20 transformation groups, a small sample for a claim of semantic invariance.","fun_headline_variants_meta":{"raw":{"variants":["Format swap flips LLM decisions 41% of the time","CSV vs text: LLM accuracy drops 53% on format change","Format switch breaks LLM workflows, fix recovers 44%","Same content, new format: LLM decisions drift 41%","LLM reliability hinges on document format, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000104,"raw_usage":{"total_tokens":885,"prompt_tokens":773,"completion_tokens":112,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":36}},"tokens_in":517,"tokens_out":112,"duration_ms":1990,"temperature":1.0,"reasoning_tokens":36,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:03:07.428323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of the paper's tasks and have multiple independent annotators verify that the TXT, Markdown, JSON, and CSV versions of each instance carry identical facts, field values, and relationships; if a nontrivial fraction of conversions are judged to shift semantics or salience, the reported violation rates conflate format effects with conversion artifacts.","supporting_citations":[],"review_version":1}