{"id":"921984f0-ec37-43c1-afd8-d50248b8e21a","arxiv_id":"2507.01810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"JSON is the most robust serialization format for small-language-model structured output extraction from clinical notes, with targeted prompting and larger models improving parseability.","lead":"This paper tests how often seven small language models produce parseable JSON, YAML, and XML outputs when extracting clinical attributes from 105 EHR notes. JSON parses most reliably, and targeted prompting sharply improves all formats, though longer and complex notes still break outputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The JSON advantage may be an artifact of pooling syntactic errors with extraction failures caused by infinite repetition; conditional parseability is never reported.","rationale":"The paper's per-row results are internally consistent, and the claim that JSON is highest in every reported model/setting is directly supported by Table 3; I credit that evidence. The most load-bearing concern is construct validity rather than external generalization. The paper's own error taxonomy separates extraction-related failures (mostly infinite repetition) from malformed-output failures, but the headline metric and the McNemar tests pool them. For end-to-end deployment, pooling is defensible, but the paper's stated purpose is to provide guidance on serialization formats. If JSON's advantage comes substantially from avoiding repetition loops, the guidance may depend on the fixed greedy decoding configuration and may not transfer to sampling, repetition penalties, or constrained decoding. This is an internal issue that can be settled by a conditional parseability analysis, which is not reported. The reader's concern about generalization beyond the fixed setup is related but distinct; hence partial agreement. The verdict remains conditional: the central claim is plausible and supported by aggregate tests, but it should be qualified until the conditional analysis is reported.","tokens_in":9918,"tokens_out":9920,"duration_ms":122581,"concrete_test":"Recompute the format comparison using only outputs from which a structured object was successfully extracted; i.e., set the denominator to |D| minus extraction-related failures (counts in Table 8) and rerun the McNemar tests on malformed-output status. Report conditional parseability rates for JSON, XML, and YAML separately for open and targeted settings. If JSON remains highest when repetition/extraction failures are removed, the headline survives; if the gap narrows or reverses, the claim should be reframed as total-output robustness rather than serialization-format robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines parseability as the fraction of outputs validated by a standard parser, so any failure to recover a structured object counts as a format failure. Section 5 and Table 8 show that a substantial part of the JSON vs. YAML/XML gap is not a syntax problem: extraction-related failures are 31 for JSON, 78 for XML, and 112 for YAML, and the overwhelming majority are infinite repetitions (31/31, 78/78, 109/112). Infinite repetition is a decoding-degeneration phenomenon (Holtzman et al., 2020), not a property of the serialization syntax. The paper never reports parseability conditional on successful extraction, so the reader cannot tell whether JSON wins because its syntax is easier for SLMs to produce or because JSON prompts happen to trigger fewer repetition loops. The McNemar tests in Table 4 are computed on total parseability, so they inherit this confound. Since the abstract frames the result as practical format selection, the confound is load-bearing: the same data could support 'JSON yields the most parseable end-to-end outputs' while saying nothing about the syntactic robustness of JSON itself; conversely, if the conditional ranking also showed JSON first, the practical guidance would be on much firmer ground.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares the parseability of JSON, YAML, and XML outputs generated by seven small instruction-tuned language models (3–14B) for open and targeted attribute-value extraction from 105 clinical notes in the EHRCon/MIMIC-III dataset. The central empirical claim is that JSON achieves the highest parseability across all models and settings, that targeted prompting and larger models improve parseability, and that longer documents and physician notes are associated with lower parseability. The authors use McNemar tests for format comparisons, chi-square tests for model size, prompt type, and note type, a point-biserial correlation for length, a logistic regression for note type and length, and a hand-coded error analysis that distinguishes extraction-related failures (mostly infinite repetition) from malformed-output syntax errors.","tokens_in":10096,"tokens_out":3962,"duration_ms":44280,"significance":"If the results hold, the paper offers practically useful guidance for selecting serialization formats and prompts in privacy-sensitive clinical NLP deployments using small local models. The reproducible experimental setup, the open models and dataset, and the detailed, format-specific error taxonomy in Appendix B are notable strengths. The paper also addresses a real gap, since most prior clinical extraction work focuses on semantic accuracy rather than syntactic robustness. However, the significance is limited by the current analysis's failure to separate decoding degeneration (infinite repetition) from true serialization-syntax robustness, and by statistical tests that ignore the repeated-measures structure of the data.","major_comments":[{"comment":"The parseability metric conflates two distinct failure modes. Table 8 shows that all 31 JSON and 78 XML 'extraction-related' errors are infinite repetitions, and 109 of 112 YAML such errors are too; these are decoding-degeneration phenomena, not serialization-syntax failures. Because parseability (Section 3.3) counts every unparsed output as a failure, the format ranking in Table 3 and the McNemar comparisons in Table 4 may reflect differential propensity of the prompts/formats to trigger repetition loops rather than the syntactic robustness of JSON/YAML/XML. Please report parseability conditional on non-repetitive output (or otherwise remove repetition failures) and state explicitly whether the JSON advantage persists on that subset.","section":"Section 5, Table 8"},{"comment":"The chi-square tests and logistic regression pool observations across models, prompt types, and formats, but the same 105 documents are measured repeatedly in all 42 conditions (7 models × 2 prompts × 3 formats). This violates the independence assumption of the tests; for example, the chi-square for prompt type (χ² = 1579.41) treats each of the thousands of outputs as independent. A mixed-effects logistic model with a random intercept for document, or cluster-robust standard errors, is needed for the document-level claims; as written, the p-values are anti-conservative. Also, 'p ≪ 0.05' is not a precise report; please provide exact p-values.","section":"Section 4"},{"comment":"The sentence 'discharge notes, though longer on average, are more parseable than nursing notes' is ambiguous relative to Figure 3, which shows nursing notes with the highest raw parseability; presumably the regression is adjusted for both length and note type, but the model specification, covariates, reference categories, and standard errors are not given. Please clarify and report the full model, including whether length is treated linearly and how repeated measures are handled.","section":"Section 4, logistic regression"}],"minor_comments":[{"comment":"The name 'YAML' is inconsistently typeset as 'Y AML' in several places, including Tables 4, 8, and in the introduction; please standardize.","section":"Throughout"},{"comment":"Report raw counts (n/N) alongside percentages; with only 105 documents, differences such as 97.8% vs. 96.9% are small and need exact counts for interpretation.","section":"Table 3"},{"comment":"Since Elnashar et al. (2025) already compared JSON and YAML (and hybrid CSV) output formats, the claim of being the 'first comparative analysis' should be qualified to 'first comparative analysis of parseability across JSON, YAML, and XML for small models in clinical note extraction' to avoid overclaiming.","section":"Section 2"},{"comment":"The prompt templates contain literal '\n' in the displayed text; ensure the appendix clearly shows the actual line breaks and that the fenced-code instructions match the regular expressions in Table 7.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a straightforward empirical comparison with a useful error taxonomy, and the central end-to-end claim (JSON is most parseable) is supported by Table 3 in every row. The main risk is interpretational: infinite repetition dominates the failure counts, yet the paper does not condition on it, so the 'structural robustness' framing is not yet justified. I also worry that the statistical tests without clustering are too optimistic; however, the data and setup are described well enough that these issues can be fixed in revision, so rejection is not warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"JSON wins on parseability, and the nice part is that the finding is bigger than the paper's caveats. Seven small models, two prompt scopes, 105 notes from EHRCon: JSON is highest in every one of the fourteen rows, and the paired McNemar tests on the JSON-vs-YAML and JSON-vs-XML contrasts are decisive. As a practical benchmark for choosing a serialization format in local clinical NLP, this is genuinely new—prior work like Elnashar et al. ran GPT-4o and focused on semantics, not SLM parseability.\n\nThe paper does a lot right. The experimental setup is transparent, the error analysis is concrete (unquoted vitals, YAML alias collisions, XML tag-name violations), and the limitations section is honest. The consistent trend across vendors and sizes is evidence the JSON advantage is not a single-model fluke.\n\nThe soft spots are real but mostly addressable. The dataset is 105 documents, and the only decoding configuration is greedy; no code is released, so reproducing the exact numbers is harder than it should be. The chi-square and logistic-regression analyses ignore the repeated-measures structure (same documents across models and settings), and reporting p as 'much less than 0.05' is sloppy. The 'model size' grouping also confounds size with vendor and architecture.\n\nOn the stress-test claim: the concern that infinite repetition loops, not syntax, drive JSON's advantage doesn't survive contact with Table 8. Extraction/repetition failures are 31 JSON, 78 XML, 112 YAML, but total parse failures are roughly 130, 428, and 538, so malformed-output errors are the bulk of the gap. Conditional parseability would make the paper cleaner, but it would not change the ranking.\n\nWho should read this: anyone deploying SLMs for structured extraction and anyone designing benchmarks that conflate generation stability with syntax. It deserves a serious referee. I'd recommend conditional acceptance, asking for exact p-values and confidence intervals, repeated-measures-aware tests, and code.","headline":"A useful and honest benchmark paper; JSON's parseability advantage over YAML and XML for small clinical LLMs is real, and the methodological caveats are fixable.","tokens_in":10618,"tokens_out":4473,"would_cite":true,"duration_ms":43397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small language models asked to pull attributes from clinical notes produce parseable JSON significantly more often than YAML or XML, and targeted prompts plus larger models push parseability higher.","keywords":["structured output","parseability","small language models","clinical notes","JSON","YAML","XML","attribute-value extraction"],"falsifier":"A concrete check: rerun the seven models on the same 105 notes at temperature 0.7 with a repetition penalty, or with grammar-constrained decoding, and compare parseability rates; if XML or YAML matches or beats JSON under either condition, the claim that JSON is consistently the most parseable format does not generalize. A second check: repeat the protocol on a different clinical corpus of comparable size and see whether YAML's open-setting weakness and JSON's lead both reproduce.","tokens_in":1776,"feed_emoji":"🩺","tokens_out":4617,"duration_ms":112852,"temperature":0.7,"pith_summary":"This paper asks which serialization format—JSON, YAML, or XML—small language models can actually produce in a form a standard parser accepts, in the task of open attribute-value extraction from clinical notes. On 105 de-identified notes from the EHRCon/MIMIC-III collection, tested with seven open-weight instruction-tuned models ranging from 3 billion to 14 billion parameters, the paper finds that JSON is consistently the most parseable format, with every pairwise difference statistically significant. It also finds that targeted prompts (asking for medications, symptoms, or demographics) improve parseability in every format, that larger models are more structurally robust, and that longer documents and physician notes are more likely to fail. The practical point is that format choice and prompt design can be decided from evidence rather than habit when deploying small models in privacy-sensitive clinical settings.","feed_headline":"JSON beats YAML and XML in small-model clinical extraction","feed_subtitle":"On 105 notes with 7 small models, JSON parsed consistently highest; targeted prompts and larger models closed most gaps.","key_machinery":"The load-bearing object is the parseability rate $\\rho(D) = n_v / |D|$, defined as the fraction of documents whose model outputs a standard parser accepts without manual correction. The argument works by generating outputs with seven small instruction-tuned models under a fixed greedy decoding configuration, extracting structured content with format-specific regular expressions when direct parsing fails, and then comparing parseability across formats with paired McNemar tests and chi-squared tests. The error analysis supplies the mechanism behind the ranking: JSON's failures concentrate in a few quotable-string violations, whereas YAML and XML accumulate more varied and harder-to-repair structural errors, and infinite repetition blocks break the extraction step itself.","core_discovery":"The central claim is that JSON outperforms both YAML and XML on parseability—the fraction of outputs a standard parser accepts without manual correction—across every tested model and both extraction scenarios. The strongest single result is the paired McNemar tests: JSON versus YAML and JSON versus XML are both significant at $p \\ll 0.05$, with JSON ahead in all 14 model-by-setting rows of Table 3. The paper also establishes that parseability is not a fixed property of a format: targeted prompts raised average parseability substantially, especially for YAML (open YAML parseability across models ranged 23.4–52.3%, while targeted YAML ranged 75.1–97.8%), and larger models failed less often, though the effect sizes were modest (Cramér's $V = 0.11$ for model size, $0.42$ for prompt type). The failure analysis attributes most extraction-related failures to infinite repetition and most malformed outputs to format-specific syntax violations, such as unquoted numerals with slashes in JSON, unescaped reserved characters in XML, and indentation or alias problems in YAML.","pith_inferences":["The paper fixes greedy decoding and free-form generation; a natural extension is grammar-constrained decoding, where XML's rigid nesting might close much of the gap with JSON, a direction the paper itself suggests but does not test.","Parseability treats every failure as equal, but downstream cost differs: a JSON unquoted number is often repaired with one regex, while a YAML indentation collapse can corrupt a whole block, so a cost-weighted robustness metric might shift the ranking.","The interaction between prompt type and error type (extraction errors made up 45.5% of targeted failures but only 2.4% of open failures) suggests repetition arises when the model is given a short, category-specific list task, which could be tested by varying list length or instruction phrasing.","Because only 105 notes from one corpus were used, the paper implicitly assumes EHRCon represents the range of clinical narrative styles; testing on a second corpus with different note templates would show whether the format ranking survives."],"forward_implications":["Default to JSON when asking a small model to return structured clinical extractions; it gives the highest chance of parseable output in both open and targeted settings.","If YAML must be used, pair it with targeted prompts and a larger model; open-ended YAML extraction is the least reliable configuration measured.","For longer and denser notes, such as physician notes with vitals and units, expect lower parseability and plan stricter prompting or post-processing that quotes values.","Address infinite repetition explicitly, for example with repetition penalties or constrained decoding, since it is the dominant extraction-level failure mode for medium and small models under targeted prompts.","Report parseability separately from semantic accuracy in structured extraction evaluations, because a syntactically invalid output fails downstream use regardless of content."],"supporting_citations":[{"why":"Supplies the EHRCon dataset of 105 clinical notes from three note types that all experiments are run on.","marker":"Kwon et al., 2025"},{"why":"Describes MIMIC-III, the critical care database from which EHRCon is derived.","marker":"Johnson et al., 2016"},{"why":"Provides the neural-text-degeneration concept of infinite repetition that dominates the paper's extraction-related error category.","marker":"Holtzman et al., 2020"},{"why":"Establishes small language models as few-shot learners, motivating the paper's selection of SLMs.","marker":"Schick and Schütze, 2021"},{"why":"Prior work comparing JSON, YAML, and CSV prompt styles with GPT-4o that the paper extends to small models and parseability.","marker":"Elnashar et al., 2025"},{"why":"Documents the trade-off between format constraints and reasoning flexibility, used to frame why parseability gains may not be free.","marker":"Tam et al., 2024"},{"why":"Defines open information extraction, the open-ended task formulation the paper evaluates.","marker":"Etzioni et al., 2008"}],"fun_headline_variants":["JSON beats YAML and XML for small-model parseability","Small models parse JSON best on clinical notes","Clinical extraction: JSON most parseable for small LLMs","Targeted prompts shore up YAML and XML parsing","Larger small models parse structured output more reliably"],"cache_read_input_tokens":12800,"weakest_assumption_plain":"The format ranking could shift if any of the fixed test conditions change—the exact prompt wording, greedy decoding with no sampling, the specific 105-note corpus, or the standard Python parsers—so the paper's 'JSON always wins' claim is about this setup, not about all structured generation.","fun_headline_variants_meta":{"raw":{"variants":["JSON beats YAML and XML for small-model parseability","Small models parse JSON best on clinical notes","Clinical extraction: JSON most parseable for small LLMs","Targeted prompts shore up YAML and XML parsing","Larger small models parse structured output more reliably"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3009,"prompt_tokens":864,"completion_tokens":2145,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":2070}},"tokens_in":480,"tokens_out":2145,"duration_ms":17985,"temperature":1.0,"reasoning_tokens":2070,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:41:30.630581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: rerun the seven models on the same 105 notes at temperature 0.7 with a repetition penalty, or with grammar-constrained decoding, and compare parseability rates; if XML or YAML matches or beats JSON under either condition, the claim that JSON is consistently the most parseable format does not generalize. A second check: repeat the protocol on a different clinical corpus of comparable size and see whether YAML's open-setting weakness and JSON's lead both reproduce.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EHRCon dataset of 105 clinical notes from three note types that all experiments are run on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work comparing JSON, YAML, and CSV prompt styles with GPT-4o that the paper extends to small models and parseability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines open information extraction, the open-ended task formulation the paper evaluates."}],"review_version":1}