{"id":"e9a6196f-9a70-4b2f-9ab2-9a2a90aefe13","arxiv_id":"2501.15000","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MDEval scores Markdown Awareness as the normalized edit distance between a model's HTML-tagged output and a GPT-4o rewrite, and reports human-alignment accuracy of 84.1% when ties are excluded.","lead":"This paper introduces MDEval, a benchmark that measures how well large language models format their chatbot answers as Markdown, by comparing each output to a GPT-4o rewrite of itself. It matters because clean output structure is a practical but under-measured part of LLM quality for web products.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline human-alignment accuracy of 84.1% is computed after skipping ties; including ties (Table 4) drops MDEval to 63.4%, below P-LLM's 65.4%, so the claimed 'large margin' over existing methods is unsupported.","rationale":"We focused on the empirical basis for the central claim rather than the design assumption. The paper itself provides both a ties-skipped accuracy (84.1%) and a with-ties accuracy (63.4%) for the same methods. Because ties are 20–38% of all judgments, the choice to headline the ties-skipped number is consequential: on the full data, MDEval is not the best method, and its margin over P-LLM disappears. This directly contradicts the abstract's 'outperforming existing methods by a large margin.' While the Spearman correlation is still favorable, the accuracy claim is the more concrete and visible number. The reader's stated weakest assumption about GPT-4o's rewrite as reference is related but more speculative; the with-ties result is a concrete, in-paper observation that the metric does not generalize to all human preferences. A revised paper should report the full-data accuracy as primary, or justify why ties should be excluded, and soften the 'large margin' language. The fine-tuning claim also depends on the metric's validity; if the metric is biased toward GPT-4o style, the improved scores after fine-tuning may partly reflect learning to imitate that style. Therefore, we maintain the conditional verdict.","tokens_in":16611,"tokens_out":7038,"duration_ms":63429,"concrete_test":"Re-analyze the human alignment dataset A from Section 3.2 without excluding ties, using a tie-aware accuracy that counts exact agreement with Eq. (6) and, separately, a soft score (e.g., 1 for correct winner, 0.5 for tie predicted and tie observed). Compare MDEval against P-LLM with bootstrap confidence intervals. If MDEval does not significantly exceed P-LLM on the full data, the headline 'large margin' claim must be revised and the benchmark's validity argument needs additional support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 and Table 2 report MDEval's human-alignment accuracy as 84.1%, with the caveat 'ties are skipped.' Table 4 in Appendix B.2 presents the same methods with ties included: MDEval's accuracy is 63.4%, while P-LLM (the pure GPT-4o judge) reaches 65.4%. Figure 4 shows ties account for 20–38% of all pairwise judgments, so the excluded data are substantial. The abstract's claim that MDEval 'outperforms existing methods by a large margin' therefore rests entirely on the ties-skipped subset. On the full data, MDEval is not the best method, and its Spearman advantage over P-LLM is small (0.791 vs 0.779). Since the central evidence for validity is human alignment, this selective reporting is a load-bearing problem: the edit-distance-to-GPT-4o-rewrite metric does not show superior agreement with human preferences once ties are considered. The reader's concern about the GPT-4o reference is related: the metric may be capturing similarity to GPT-4o's rewriting style rather than general readability, and the tie-heavy comparisons are precisely where that approximation fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MDEval, a benchmark for measuring the 'Markdown Awareness' of LLMs, defined as the structural quality of Markdown formatting in generated responses. The pipeline generates a response from a target LLM (Phase 1), rewrites that response using GPT-4o with a fixed prompt to create a model-specific reference (Phase 2), converts both outputs to HTML and extracts tags (Phase 3), and computes a normalized Levenshtein edit distance between tag sequences as the score (Phase 4, Eq. 1). The authors construct a 20K-instance dataset in English and Chinese across 10 subjects, evaluate 9 LLMs, and validate MDEval through a human pairwise-comparison study with Elo ratings. They report a record-level human-alignment accuracy of 84.1% and Spearman correlation 0.791 (Table 2), claiming to outperform existing methods by a large margin. They also show that QLoRA fine-tuning on a subset of the dataset improves Baichuan2-13B-chat-v1's MDEval score from 0.171 to about 0.73, which they describe as approximating GPT-4o-level performance.","tokens_in":16854,"tokens_out":3047,"duration_ms":26911,"significance":"If the validity claims hold, MDEval would be a useful, low-cost benchmark for a dimension of LLM output quality that is important for web chatbots and largely ignored by prior benchmarks. The paper provides a new dataset, an open-source implementation, and a human-alignment evaluation system, all of which are valuable community resources. The core idea of comparing HTML-tag edit distance rather than raw surface text is sensible and interpretable. However, the paper's central evidentiary claim—that MDEval aligns with human preferences substantially better than existing methods—is weakened by the selective reporting of ties-skipped accuracy, and the use of GPT-4o as both the rewrite judge and one of the evaluated models raises a circularity risk that is only partially addressed by the human study. The fine-tuning result is also overstated relative to the reported GPT-4o score. These issues are fixable but require additional analysis and experiments.","major_comments":[{"comment":"The headline human-alignment result of 84.1% accuracy for MDEval is computed on the subset of pairwise comparisons where ties are skipped. Appendix B.2, Table 4, which includes ties, reports MDEval accuracy as 63.4%, below P-LLM's 65.4%. Figure 4 shows that ties constitute 20–38% of judgments, so the excluded data are substantial and non-random. The abstract and Section 5.3 claim that MDEval 'outperforms existing methods by a large margin,' but this conclusion is not supported on the full dataset. Please report the ties-included results as the primary analysis or provide a principled justification for excluding ties, and qualify the 'large margin' claim accordingly.","section":"§5.3, Table 2 vs. Appendix B.2, Table 4"},{"comment":"The reference used for scoring is a GPT-4o rewrite of the target model's own output, produced with a single fixed prompt at temperature 1. Since GPT-4o is also one of the evaluated models (Table 1), the metric may partly measure agreement with GPT-4o's particular rewriting style rather than general Markdown readability. The human-alignment study provides external grounding, but it does not eliminate this concern because the human study uses the same pairwise framework and the metric is still anchored to GPT-4o. Please report robustness checks with multiple judge LLMs (e.g., Claude or Gemini as the rewriter), and quantify the variance of scores due to the stochastic rewriting (temperature 1, Appendix B.5).","section":"§3.1, Phase 2 and Eq. (1)"},{"comment":"The abstract and Section 5.5 claim that fine-tuning on the proposed dataset enables less performant open-source models to achieve 'comparable performance to GPT-4o.' However, Figure 7 shows the fine-tuned Baichuan2-13B-chat-v1 reaches an MDEval score of about 0.73, while Table 1 lists GPT-4o at 0.865 and GPT-4o-mini at 0.830. This is not comparable to GPT-4o. Additionally, the fine-tuned model is only evaluated with the MDEval metric, which itself relies on GPT-4o as the rewriter; no human evaluation of the fine-tuned model's outputs is provided. Please revise the claim and add human or at least multi-judge evaluation of the fine-tuned model.","section":"§5.5, Figure 7"},{"comment":"The record-level accuracy indicator in Eq. (6) treats a human tie as correct only if the two MDEval scores are exactly equal. Since MA scores are continuous (normalized edit distances), exact equality essentially never occurs, so every human tie is automatically counted as an error when ties are included. This explains the large drop from 84.1% to 63.4% and makes the ties-included accuracy a harsh and arguably inappropriate measure. A tolerant tie-handling rule (e.g., treating score differences below a threshold as ties) or a soft agreement metric should be defined and reported.","section":"§3.2, Eq. (6)"}],"minor_comments":[{"comment":"Typo: 'textural' should be 'textual' in the sentence about content bias.","section":"§3.1"},{"comment":"Typo: 'fine-turning' should be 'fine-tuning'.","section":"§5.5"},{"comment":"The phrase 'count the number of Markdown enumerate' should likely be 'count the number of Markdown elements'.","section":"§5.1"},{"comment":"The heading 'QR5' appears to be a typo for 'RQ5'.","section":"Appendix B.4"},{"comment":"The caption says 'at an early snapshot of A', but the text later states the tie ratio 'stays relatively constant as the amount of collected data grows.' Please clarify whether Figure 4 shows the final dataset or an early snapshot, and report the total number of pairwise judgments collected.","section":"Figure 4"},{"comment":"The claim that outputs at temperature 1 are 'stable generally' is surprising and should be supported by repeated-generation statistics, since temperature 1 typically implies high stochasticity.","section":"Appendix B.5"},{"comment":"The dataset is described as ground-truth-free, but Section 4.2 states that fine-tuning uses 'the response generated by the most advanced LLM' as ground truth. Please clarify that the evaluation dataset is ground-truth-free while the fine-tuning set uses a model-generated reference.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and under-explored problem for the Web Conference audience, and the released resources are a plus. My main concern is the selective reporting of the ties-skipped accuracy, which changes the ranking of methods when ties are included; this is a load-bearing issue for the paper's central claim. The circularity of using GPT-4o as both judge and evaluated model is also a risk that needs to be addressed with additional experiments rather than only discussion. I believe the work is potentially salvageable with major revisions, including re-analysis of human alignment with proper tie handling and multi-judge robustness checks. I would not recommend rejection at this stage, but the current version overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is the first benchmark I've seen that actually targets Markdown output structure as a measurable capability. The dataset (20K prompts, two languages, ten subjects) is real, the pipeline (generate, rewrite with GPT-4o, HTMLify, edit distance) is simple and reproducible, and they open-source everything. The fine-tuning result, that a small open model can be pushed from 0.171 to 0.73 MDEval score with only 1600 SFT instances, is a genuine and useful finding for anyone building chatbot systems. I'd send this to a serious referee without hesitation.\n\nThe soft spot is exactly what the stress-test note says. The abstract and Section 5.3 report 84.1% accuracy and claim to 'outperform existing methods by a large margin,' but that number is computed after skipping ties. Appendix B.2, Table 4, shows the with-ties accuracy: MDEval drops to 63.4%, which is below P-LLM's 65.4%. Ties are 20–38% of all human judgments, so this isn't a minor detail. The Spearman correlation stays decent (0.791 vs 0.779), but the accuracy claim in the abstract is simply not supported by the full data. The authors do disclose the tie-inclusive table in an appendix, so it's not a hidden fabrication, but the main text misrepresents the metric's actual advantage.\n\nRelated concern: the reference is a GPT-4o rewrite of the model's own output, and GPT-4o is itself one of the nine evaluated models. That creates a bias toward models that happen to match GPT-4o's formatting style. The human alignment study partially grounds the metric, but the tie-heavy cases are precisely where the edit-distance-to-GPT-4o approximation seems to fail. Also, the fine-tuning claim 'comparable to GPT-4o' overstates the numbers: 0.73 vs 0.865 is close-ish, but 'comparable' implies something stronger than what they show.\n\nBottom line: the core idea is sound and the dataset is a real contribution. But the paper needs a revision that either defends the ties-skipped protocol (e.g., arguing ties are uninformative for accuracy) or, more honestly, reports with-ties accuracy as the headline and tones down the 'large margin' language. The Spearman advantage over P-LLM is small but real, so the metric may still be useful. I'd recommend: accept in principle, but require the authors to re-frame the human-alignment claims and report all results with and without ties on equal footing.","headline":"A genuinely new benchmark for Markdown formatting, but the headline accuracy claim is cherry-picked (ties skipped) and needs an honest rewrite before the numbers can be trusted.","tokens_in":17429,"tokens_out":1273,"would_cite":false,"duration_ms":13028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Markdown Awareness is measurable: MDEval matches humans on 84.1% of pairwise formatting comparisons and lifts weaker models by fine-tuning.","keywords":["Markdown Awareness","LLM evaluation benchmark","web chatbot readability","human alignment","Levenshtein distance","structure-oriented metric","supervised fine-tuning"],"falsifier":"Re-run the benchmark with a different judge LLM or a different rewrite prompt; if model rankings shift materially, the metric is judge-dependent. Or run a preference study in which humans choose between a model's original response and its GPT-4o rewrite: the paper's alignment claim predicts the rewrite should win most pairings, so frequent human choices for the original or a third style would falsify the claim that the edit-distance proxy captures human readability.","tokens_in":16367,"feed_emoji":"📝","tokens_out":9074,"duration_ms":70999,"temperature":0.7,"pith_summary":"Markdown Awareness is the quality of how a large language model structures a response with Markdown — headings, lists, bold, code, and math — when the prompt does not ask for formatting. The paper argues this is an overlooked ability with direct impact on web-chatbot readability, and introduces MDEval, a 20,000-prompt benchmark across ten subjects in English and Chinese to measure it. MDEval scores each response by having a fixed judge model rewrite it into a well-structured reference, converting both versions into HTML, and computing the normalized edit distance between the extracted tag sequences. The authors report 84.1% agreement with human pairwise preference and a Spearman correlation of 0.791, and show that fine-tuning a 13B open-source model on the curated data raises its Markdown Awareness to a level the paper describes as comparable to GPT-4o.","feed_headline":"Benchmark matches humans 84% on chatbot formatting","feed_subtitle":"MDEval grades Markdown awareness by edit distance to a GPT-4o rewrite, and fine-tuning lifts small models.","key_machinery":"The HTMLification-and-edit-distance pipeline is the load-bearing mechanism: each response and its judge-written reference are rendered to HTML, the tags are extracted as a string, and the Markdown Awareness score is $$1 - \\text{editDistance}(r, \\hat{r})/\\max(\\text{len}(r), \\text{len}(\\hat{r})).$$ This recasts \"how well structured is this answer\" as \"how few tag-operations separate it from a high-quality rewrite of the same content,\" giving an interpretable, content-free measure. A secondary mechanism is the Elo-rating crowdsourcing harness, which provides the human ground truth the metric is validated against.","core_discovery":"Markdown Awareness, as defined here, is a structure-oriented, ground-truth-free score: for a given prompt and model, the model's response is rewritten on the fly by a leading LLM into an equivalent but well-structured Markdown version, and the two are compared purely as HTML tag strings. Custom <math> tags preserve LaTeX equations that standard Markdown parsers drop. The score is 1 minus the normalized Levenshtein distance, so 1 means the model's own formatting already matches the rewrite. The paper's central evidence is that this score tracks humans: in a pairwise Elo-based crowdsourcing study, MDEval reaches 84.1% record-level accuracy and 0.791 Spearman correlation, beating the LLM-judge baselines P-LLM and R-LLM and the rule-based D-Rule. The benchmark also doubles as a training set, with supervised fine-tuning lifting Baichuan2-13b-chat-v1 from 0.171 to about 0.73.","pith_inferences":["If formatting taste is plural, a single judge model and prompt may bake in one style; swapping the judge or averaging several judges would reveal how much of the ranking is an artifact of GPT-4o's own formatting preferences.","The edit distance treats all tag edits equally, so a missing heading and a reordered list cost the same; a weighted or tree-aware tag distance might track human readability even more closely while keeping the pipeline interpretable.","The paper validates against preference, not comprehension; a reading-time or recall experiment on rendered pages would test the stated cognitive-load motivation directly.","The ground-truth-free, model-dependent reference design is portable to other style-oriented qualities, such as tone, citation formatting, or table use, where fixed ground truths are similarly unavailable."],"forward_implications":["Model rankings for chatbot readability can be obtained automatically, without human judges or content-specific ground truth, at the cost of one rewrite call and one string comparison per response.","The dataset doubles as a supervised fine-tuning resource, so weaker open models can be lifted toward the formatting quality of much stronger models.","Because each reference is a rewrite of the model's own words, the benchmark sidesteps the problem that no single canonical Markdown answer exists for a prompt.","Since scoring works on HTML tags, the same pipeline can be adapted to other structured output formats such as plain HTML, wiki markup, or table-heavy layouts.","The resulting rankings differ from general leaderboards: Deepseek-v2-chat beats GPT-4o, and Llama-3.1-8b beats Claude-3.5-sonnet, suggesting Markdown Awareness is a distinct capability."],"supporting_citations":[{"why":"Supplies the pairwise human-comparison protocol and Elo parameter settings (d=400, K=10) used in the alignment study.","marker":"[6]"},{"why":"G-Eval is the main model-based baseline and the source of the accuracy/correlation methodology for human alignment.","marker":"[21]"},{"why":"BLEU represents the statistical content-overlap metrics that the paper argues are unsuitable for structure-oriented Markdown evaluation.","marker":"[27]"},{"why":"ROUGE is another content-oriented statistical baseline used to position the proposed structure-oriented metric.","marker":"[7]"},{"why":"Documents the common practice of stripping Markdown from training text, motivating the fine-tuning enhancement.","marker":"[10]"},{"why":"Supplies the 'quality is all you need' principle behind curating the SFT set from the strongest model's outputs.","marker":"[30]"},{"why":"Provides the QLoRA fine-tuning framework used in the fine-tuning experiments.","marker":"[40]"},{"why":"The Elo rating formula underlies the human-preference rankings used to validate MDEval.","marker":"[32]"}],"fun_headline_variants":["Markdown awareness benchmark matches human judgment 84%","MDEval: ground-truth-free scoring of LLM Markdown formatting","Fine-tuning on MDEval lifts small LLMs to GPT-4o Markdown skill","New benchmark gauges LLM Markdown readability with edit distance","20K-prompt benchmark tracks Markdown awareness with 84% human accord"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement assumes that GPT-4o's rewrite of a response, produced with a single fixed prompt, is the correct reference for well-structured Markdown, and that tag edit distance to that reference is what humans mean by readability; if either fails, the rankings may reward mimicry of one formatting style instead of general readability.","fun_headline_variants_meta":{"raw":{"variants":["Markdown awareness benchmark matches human judgment 84%","MDEval: ground-truth-free scoring of LLM Markdown formatting","Fine-tuning on MDEval lifts small LLMs to GPT-4o Markdown skill","New benchmark gauges LLM Markdown readability with edit distance","20K-prompt benchmark tracks Markdown awareness with 84% human accord"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1844,"prompt_tokens":983,"completion_tokens":861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":765}},"tokens_in":599,"tokens_out":861,"duration_ms":11668,"temperature":1.0,"reasoning_tokens":765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:43:00.220587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with a different judge LLM or a different rewrite prompt; if model rankings shift materially, the metric is judge-dependent. Or run a preference study in which humans choose between a model's original response and its GPT-4o rewrite: the paper's alignment claim predicts the rewrite should win most pairings, so frequent human choices for the original or a third style would falsify the claim that the edit-distance proxy captures human readability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise human-comparison protocol and Elo parameter settings (d=400, K=10) used in the alignment study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Elo rating formula underlies the human-preference rankings used to validate MDEval."}],"review_version":1}