{"id":"765d3701-87f6-41e5-b6a7-1d661ca2f0c6","arxiv_id":"2504.14165","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A treebank-rule-guided self-correction method improves LLM constituency parsing F1 by up to 10.12 points on PTB, CTB5, and MCTB across multiple LLMs.","lead":"This paper analyzes why large language models produce poor syntactic parse trees and proposes a self-correction method that retrieves grammar rules from existing treebanks to guide the model's corrections. The method improves parsing F1 by about 10 points on English and Chinese benchmarks without retraining, with larger gains for stronger models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 and Figure 5 report irreconcilable unknown-rule accuracy for GPT-4 on PTB (37.14% vs 6.64%), so the paper's key rule-statistics evidence for the error detector is internally inconsistent and needs reconciliation.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being that rule absence from the treebank indicates error, citing Table 2's 37.14% unknown-rule accuracy for GPT-4 on PTB as evidence that the detector is noisy. My stress-test identifies a more fundamental evidentiary problem: the paper's own quantitative characterization of rule accuracy is internally inconsistent. Table 2 and Figure 5(a) give mutually incompatible numbers for the same model and dataset. If Table 2 is correct, the reader's concern about detector noise is well-founded; if Figure 5 is correct, the detector is actually quite precise and the reader's concern is largely unfounded. The paper cannot support both. This inconsistency is directly load-bearing because the method's error-detection step (Section 3.2 step 1) is the foundation of the entire correction pipeline, and the paper's motivating analysis (Section 2.2) is built on the same rule-statistics numbers. Without code or a precise definition of rule accuracy, a reader cannot decide which figure is trustworthy, and therefore cannot determine whether the method works as described or whether the reported F1 improvements arise from a different mechanism. The F1 improvements themselves are still reported uniformly positive across models and datasets, and the random-selection and POS-searching comparisons provide some internal control, so I do not think the central claim is overturned. However, the inconsistency strengthens the need for the conditions already identified by the reader (code release, clearer evaluation details), and adds a specific new condition: reconcile the rule-statistics discrepancy. I therefore keep the verdict at CONDITIONAL rather than moving it, and I partially agree with the reader because the noisy-detector concern may or may not hold depending on which table is correct.","tokens_in":15429,"tokens_out":25728,"duration_ms":238874,"concrete_test":"Recompute the rule statistics for GPT-4 on PTB using the authors' exact extraction and matching scripts: for each predicted rule (parent label plus ordered child labels), determine whether it occurs in the gold tree at the same span; then compute known/unknown status against the PTB train treebank and compare the resulting accuracy figures to Table 2 and Figure 5. Also run the same computation for one smaller model (e.g., GPT-3.5) to check whether the discrepancy is isolated. If the recomputed unknown-rule accuracy is close to 37.14%, Figure 5 is wrong and the reader's noisy-detector concern stands; if it is close to 6.64%, Table 2 is wrong and the detector is much more precise than stated. In either case, the authors must release the rule-extraction/evaluation code and correct the inconsistent exhibit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 and Figure 5(a) describe the same quantity for GPT-4 on PTB: the accuracy of known and unknown grammar rules in the base parsing results. Table 2 reports known-rule accuracy 81.44% and unknown-rule accuracy 37.14%, with 20,291 known and 19,875 unknown rules. Under the natural span-level definition of rule correctness (the predicted parent+children rule matches the gold tree at the same span), these rates imply 16,526 correct known rules and 7,381 correct unknown rules. Figure 5(a), using exactly the same known/unknown totals (20,291 and 19,875), shows only 15,897 correct known rules and 1,320 correct unknown rules, corresponding to known accuracy 78.3% and unknown accuracy 6.64%. Both cannot be correct. This discrepancy is not a minor rounding issue: the unknown-rule accuracy differs by a factor of 5.6. The paper's Section 2.2 uses the high unknown-rule accuracy to motivate the central thesis that LLMs lack known rules, while the noisy-detector premise in Section 3.2 step 1 (rule absence implies error) is assessed by the reader using the same Table 2 number. If the true figure is ~6.6%, the rule-absence detector is far more precise than the reader's analysis assumes; if it is 37.1%, Figure 5 is erroneous. The paper does not define 'accuracy' precisely, does not provide code, and does not reconcile the two presentations. This internal inconsistency undermines the reliability of the empirical analysis that justifies the method's design and raises uncertainty about whether the implemented error-detection stage behaves as described. Since the reported F1 gains depend on that implementation, resolving this discrepancy is a precondition for trusting the central claim's interpretation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies why LLMs underperform on constituency parsing and proposes a training-free self-correction method. It first analyzes parsing outputs by comparing grammar rules extracted from predicted trees against treebank rules and by classifying errors into span, label, flatness, and deepness categories. It then proposes a two-stage correction: unmatch correction to align leaf words, and structure correction that detects subtrees whose rules are absent from the treebank, retrieves similar rules via LCS ranking after error-type-specific processing, and prompts the LLM to rewrite the subtree using sampled examples. Experiments on PTB, CTB5, and MCTB with LLaMA-3, Qwen-2.5-72B, DeepSeek-V3, GPT-3.5, and GPT-4 report large positive gains over the few-shot baseline, e.g., GPT-4 PTB F1 73.38 to 83.50 and CTB5 44.69 to 64.99, with consistent improvements across all model/dataset combinations.","tokens_in":15725,"tokens_out":5503,"duration_ms":48595,"significance":"If the results are reliable, the paper makes a useful contribution: a training-free method for improving LLM constituency parsing that is simple, model-agnostic, and appears to generalize across English and Chinese. The large and consistent gains, especially the recall improvements, plus the random-selection control in Section 4.6, provide evidence that the retrieved rules and examples carry information beyond simply adding more examples. The analysis of rule statistics and error types is also a useful diagnostic. However, the reliability of the central empirical claims is currently weakened by an internal inconsistency in the rule statistics (Table 2 versus Figure 5), by apparent selection of hyperparameters on the PTB test set, and by a contamination check performed only on LLaMA-8B rather than on the closed models used for the headline results. These issues are fixable, so the paper is not beyond repair.","major_comments":[{"comment":"The paper reports irreconcilable numbers for the accuracy of unknown rules for GPT-4 on PTB. Table 2 reports 81.44% known-rule accuracy over 20,291 known rules and 37.14% unknown-rule accuracy over 19,875 unknown rules; under the natural span-level definition of rule correctness used in Figure 5(a), these rates imply 16,526 correct known rules and 7,381 correct unknown rules. Figure 5(a), with the same totals, shows 15,897 correct known rules and 1,320 correct unknown rules, corresponding to about 78.3% and 6.64% accuracy, respectively. The unknown-rule accuracy differs by a factor of about 5.6. This is not a minor rounding discrepancy: Section 2.2 uses the high unknown-rule accuracy to motivate the claim that LLMs lack known rules, and Section 3.2 step 1 relies on rule absence as an error signal, whose precision depends directly on this quantity. The authors must define the exact rule-accuracy computation, reconcile the two presentations, and ideally release the counting script or per-rule statistics.","section":"Table 2, Figure 5(a), Section 2.2"},{"comment":"The contamination check is performed only with LLaMA-8B, yet the main results include GPT-3.5 and GPT-4, and Table 1 uses LLaMA-70B, Qwen-72B, and DeepSeek-V3. The MKP analysis therefore supports a statement only about LLaMA-8B; the conclusion that \"these datasets are unlikely to have been included in the pre-training data\" overreaches when applied to the closed models. The authors should either qualify the claim to the open models actually tested or provide additional evidence for the closed models, and discuss how partial contamination would affect the interpretation of the baseline and corrected results.","section":"Appendix A.3, Section 2.1"},{"comment":"Key hyperparameters appear to be selected on the PTB test set itself. Figure 4 reports the effect of the minimum correction height h on PTB, and Appendix A.4 compares the number of retrieved rules and examples, reporting that the setting \"5 rules * 1 example\" works better, again on PTB with GPT-4. No development split is described, and no error bars or significance tests are given for any of the main results in Table 3. This makes the reported improvements vulnerable to selection bias and leaves the reader unable to judge whether the gains are statistically distinguishable from the random-selection baseline. The authors should specify a dev/test protocol, report variance across runs or example sets, or otherwise justify that the chosen settings were not tuned on the evaluation set.","section":"Figure 4, Appendix A.4, Section 4.2"}],"minor_comments":[{"comment":"The abstract contains an ungrammatical phrase, \"LLMs may stem from limitations to fully leverage grammar rules,\" and the footnote in Section 2.2 misspells \"involve\" as \"involve.\" These should be corrected.","section":"Abstract and Section 2.2"},{"comment":"The model name \"DeeSeekp-v3\" is inconsistent with \"DeepSeek-V3\" used in the text and in Table 9; please unify the spelling.","section":"Tables 1 and 2"},{"comment":"Figure 2 would be easier to read with an explicit legend or directly labeled bars, since the four error types are currently identified only through the caption and surrounding text.","section":"Figure 2"},{"comment":"There are typos in \"unknownn rules\" and \"we conducte a detailed analysis\"; these should be fixed.","section":"Section 4.5 and Appendix A.6"},{"comment":"The MKP interpretation would benefit from a brief explanation of why a higher MKP indicates a lower likelihood of pre-training inclusion, since the phrasing is otherwise unintuitive.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"I do not see evidence of questionable author behavior, but the Table 2/Figure 5 inconsistency is severe enough that the paper should not be accepted until it is resolved. If the authors cannot release code or align the definitions behind these two presentations, I would recommend rejection. The paper would also benefit from an explicit statement of how hyperparameters were selected relative to the test sets, as this bears directly on the credibility of the reported gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The core finding is credible: a training-free loop that detects likely wrong subtrees by checking against treebank grammar rules, retrieves LCS-ranked example rules, and asks the LLM to correct itself. On PTB, GPT-4 goes from 73.4 to 83.5 F1, and the gains are consistent across English/Chinese and open/closed models. The random-selection control (Table 4) is the right experiment and shows the retrieval component matters. That's a real, useful contribution for anyone building syntax-aware pipelines on top of LLMs.\n\nThe soft spots are addressable but real. No code or error bars; the hyperparameter sweep (minimum subtree height h) is run on the PTB test set itself, which inflates the headline number by an unknown amount. The contamination check covers only Llama-8B, not GPT-3.5/GPT-4, so the leak question for the main results remains open. And the load-bearing assumption that \"rule not found in treebank => error\" is known to be noisy from the paper's own numbers.\n\nThe bigger problem is internal consistency. Table 2 and Figure 5(a) report the same quantity for GPT-4 on PTB—the accuracy of known and unknown rules in the base output—and they disagree wildly. Table 2 gives unknown accuracy 37.14% (7,381 correct of 19,875). Figure 5(a) shows only 1,320 correct unknown rules (6.64%). Section 4.5 later quotes the pre-correction unknown accuracy as 35.3%. These cannot all be right. The known-rule numbers also differ (16,526 vs 15,897). The paper never defines \"accuracy\" precisely, and the discrepancy changes the interpretation of the method: if the true unknown accuracy is about 7%, the rule-absence detector is much more precise than the reader's analysis assumed, but the paper's story about LLMs lacking known rules needs rework. This is not a rounding error; it's a factor of 5.6.\n\nWho is this for? Researchers working on LLM parsing or retrieval-augmented prompting for structured prediction. It deserves a serious referee, but it needs major revision before acceptance: reconcile the rule statistics, release code and variance numbers, and move the h selection off the test set. If they do that, this could be a solid paper.","headline":"Promising training-free self-correction for LLM parsing, but the paper's own rule statistics contradict each other by a factor of 5.6 and the main analysis needs reconciliation.","tokens_in":16354,"tokens_out":3811,"would_cite":true,"duration_ms":30799,"reading_group":"yes","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rule-guided self-correction lets LLMs recover 10–20 F1 points in constituency parsing without retraining.","keywords":["constituency parsing","large language models","self-correction","treebank grammar rules","few-shot learning","error analysis","cross-domain parsing"],"falsifier":"Take the same two-pass correction pipeline but replace the rule-absence detector with an oracle that flags exactly the subtrees that disagree with gold parses; if the F1 gain is no larger than the paper's, then error detection via treebank lookup is not what drives the improvement. A cheaper check is to count how often the subtrees flagged by rule absence actually match the gold tree.","tokens_in":15179,"feed_emoji":"🌳","tokens_out":6455,"duration_ms":57744,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models' weak constituency parsing does not require retraining: most errors trace back to the models' failure to use grammar rules that already exist in treebanks, and a training-free self-correction loop that feeds those rules back as hints can fix them. The authors support this with an error analysis that splits LLM parse mistakes into span, label, flatness, and deepness errors, and with a correction pipeline that detects suspect subtrees by looking up their parent-child rules in the treebank. On five-shot parsing, the method lifts GPT-4's F1 on the Penn Treebank from 73.38 to 83.50 and on CTB5 from 44.69 to 64.99, with positive gains for every model and dataset tested. If right, the result means that a substantial part of LLM parsing weakness is recoverable knowledge rather than a hard architectural limit.","feed_headline":"Rule-guided self-correction lifts LLM parsing by up to 20 F1","feed_subtitle":"Training-free method uses treebank rules as hints, raising GPT-4's PTB F1 from 73.38 to 83.50 and CTB5 from 44.69 to 64.99.","key_machinery":"The load-bearing mechanism is rule lookup against an existing treebank: every subtree is compressed to a rule, meaning the parent tag plus the sequence of its children's tags, and the treebank's train split is treated as the authority on which rules are valid. Absence from the treebank marks a subtree as needing correction; candidate rules are then aligned to the detected error type (label, flatness, deepness) and ranked by LCS of child-tag sequences, with frequency as tie-breaker, so that the LLM receives the most similar attested structures as in-context examples. The unmatch-correction step, which passes back targeted hints about added, omitted, split, or modified words, operates first to ensure the leaves of the predicted tree align with the original sentence.","core_discovery":"The central claim is that LLMs underperform at constituency parsing largely because they do not systematically exploit the grammar rules (parent-child subtrees) of existing treebanks, and that supplying those rules through a self-correction mechanism, without any parameter updates, yields substantial and consistent gains. The method first repairs leaf mismatches between the predicted tree and the input sentence, then walks the predicted tree top-down; for each subtree whose rule is absent from the treebank, it treats the rule as likely erroneous, transforms candidate treebank rules to match three error types (label, flatness, deepness), ranks them by longest common subsequence of child label sequences, and prompts the LLM with the top five rule examples to regenerate that subtree. The paper reports F1 gains across GPT-4, GPT-3.5, LLaMA-3-8B, LLaMA-3-70B, Qwen-2.5-72B, and DeepSeek-V3 on PTB, CTB5, and MCTB, including cross-domain settings, and argues that correcting from the top down indirectly reduces span errors as well.","pith_inferences":["Because the paper's own numbers show that many rules absent from the treebank are actually correct, the detector is a noisy proxy; the gains may come less from accurate error identification than from giving the LLM a second chance with any valid treebank examples, a hypothesis that could be tested by comparing against a detector that flags subtrees using gold parses.","The indirect reduction of span errors suggests a broader design principle for structured prediction: correcting a full constituent structure from the top down can repair boundary errors without explicit span-level supervision, which might transfer to semantic parsing or named-entity recognition where treebank-style constraints exist.","The method's value should scale with how completely the treebank covers the test domain's rules; a natural testable extension is to measure gains on domains with deliberately low rule overlap, where the paper's logic predicts smaller improvements.","Since the paper reports that about 4% of originally correct rules are turned wrong during correction, running the self-correction loop for multiple iterations could accumulate such errors; a multi-round experiment would reveal whether gains saturate or reverse."],"forward_implications":["The improvement is training-free: any LLM that can follow few-shot prompts can adopt the method without gradient updates or task-specific fine-tuning.","Recall gains are consistently larger than precision gains, indicating the method pushes LLMs toward deeper, less flat trees and directly counters their known tendency to under-parse.","The method transfers to cross-domain parsing, improving performance on dialogue, forum, law, literature, and review domains even when the treebank comes from a different genre.","All four error types decrease after correction, and span errors—the most frequent category—are reduced indirectly through the top-down structural correction process.","Selecting examples by error-specific rule ranking beats both random example selection and POS-sequence-based selection, showing that the retrieval and ranking steps carry the benefit."],"supporting_citations":[{"why":"Establishes the LLM constituency parsing setting and documents hallucinated or unmatched outputs, motivating the unmatch-correction stage.","marker":"(Bai et al., 2023)"},{"why":"Provides the prior LLM parsing approach and numbers that the paper compares against, and supplies the observation that LLMs produce flatter trees.","marker":"(Tian et al., 2024)"},{"why":"Supplies the claim that parsers tend to reuse structures seen in training data, which grounds the known-rule vs unknown-rule analysis.","marker":"(Dakota and Kübler, 2021)"},{"why":"Gives the strong non-LLM neural parser used as the performance baseline throughout the experiments.","marker":"(Kitaev and Klein, 2018)"},{"why":"Defines the Penn Treebank, the English in-domain test set and the source of English grammar rules for rule lookup.","marker":"(Marcus et al., 1993)"},{"why":"Defines the Chinese Penn Treebank 5, the Chinese in-domain test set and the source of Chinese grammar rules.","marker":"(Xue et al., 2005)"},{"why":"Provides the Multi-domain Constituent Treebank used for the cross-domain evaluation.","marker":"(Yang et al., 2022b)"},{"why":"Supports the design choice that LLMs can correct errors once the error location is given, which is exactly what the rule lookup provides.","marker":"(Tyen et al., 2023)"},{"why":"Supplies evidence that intrinsic self-correction without external feedback often fails, motivating the use of treebank rules as external feedback.","marker":"(Huang et al., 2023)"},{"why":"Provides the survey and categorization of self-correction strategies that the paper situates its method within.","marker":"(Pan et al., 2024)"}],"fun_headline_variants":["No training, just treebank hints: LLM parsing up 20 F1","Treebank rules as hints: LLM self-correction lifts parsing F1","Grammar-guided self-correction boosts LLM parsing without retraining","Self-correction with treebank rules yields 20 F1 gain for LLM parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline treats a grammar rule that is absent from the treebank as an error, even though the paper's own analysis shows many unseen rules are correct; if that equation is wrong often enough, the detector both flags valid subtrees and misses genuine errors.","fun_headline_variants_meta":{"raw":{"variants":["No training, just treebank hints: LLM parsing up 20 F1","Treebank rules as hints: LLM self-correction lifts parsing F1","Grammar-guided self-correction boosts LLM parsing without retraining","Self-correction with treebank rules yields 20 F1 gain for LLM parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001272,"raw_usage":{"total_tokens":5210,"prompt_tokens":960,"completion_tokens":4250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":4167}},"tokens_in":576,"tokens_out":4250,"duration_ms":27995,"temperature":1.0,"reasoning_tokens":4167,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:54.817327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same two-pass correction pipeline but replace the rule-absence detector with an oracle that flags exactly the subtrees that disagree with gold parses; if the F1 gain is no larger than the paper's, then error detection via treebank lookup is not what drives the improvement. A cheaper check is to count how often the subtrees flagged by rule absence actually match the gold tree.","supporting_citations":[],"review_version":1}