{"id":"5c9f7097-8769-4357-9bb5-921c886ea71d","arxiv_id":"2607.02307","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"CCG-directed symbolic backend yields 75.9% LF exact match on SLOG versus 70.8% for AM-Parser, with +29.9pp gains isolated to all five position-shift categories.","lead":"The paper redesigns a symbolic parser backend using CCG directed types to encode directionality, achieving higher exact match on SLOG directional test categories than the prior AM-Parser under the same encoder. A smart generalist might read it to see how explicit structural direction in hybrid systems can shift generalization bottlenecks from symbolic to neural layers.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Comparison confounds directionality with CKY parser + linear decoder differences; no ablation isolates directed types as the cause of +29.9pp gains.","rationale":"The reader's weakest assumption directly matches the load-bearing gap: lack of isolation between directed types and other implementation choices. This concern is internal to the experimental design and does not rely on external consensus. Adding the proposed ablation would either confirm or refute the attribution, moving the verdict from UNVERDICTED to CONDITIONAL or ACCEPT depending on outcome.","tokens_in":1726,"tokens_out":354,"duration_ms":15942,"concrete_test":"Re-implement the CCG system with undirected types (identical CKY + linear decoder, same 30K parameters, same BERT-base) and evaluate on the 5 position-shift categories; if the +29.9pp gap versus AM-Parser shrinks below 10pp, directionality is not the isolated driver.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the performance gap on the 5 position-shift categories is caused by CCG directed types rather than other system differences. The CCG backend is a deterministic CKY parser plus single linear decoder (30K parameters); AM-Parser uses an undirected AM algebra. The abstract states gains are \"highly directional\" under the same BERT-base encoder, but does not report a controlled comparison that holds the parser, decoder, and training fixed while toggling only directionality. SLOG groupings are assumed to separate direction from recursion depth and lexical factors, yet no verification of orthogonality (e.g., category-wise recursion statistics) is described. If the parser architecture itself favors position-shift tasks, the attribution to directionality does not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that directionality in symbolic representations, via CCG directed types, is key to structural generalization on position-shift tasks. It redesigns the backend as a deterministic CKY parser plus single linear decoder (30K parameters) and reports 75.9±6.4% LF exact match under BERT-base (vs. AM-Parser's 70.8±4.3%), with +29.9pp gains on all 5 position-shift SLOG categories and complementary losses on recursive-depth categories. Upgrading the encoder to DeBERTa-v3-large reaches 90.7±4.9%, shifting the bottleneck from the symbolic layer to the neural encoder.","tokens_in":1906,"tokens_out":565,"duration_ms":21340,"significance":"If the attribution of gains specifically to directed types holds after controlling for parser and decoder differences, the result would be significant for compositional generalization work. It supplies a concrete, low-parameter demonstration that directed types address position-shift failures that undirected AM algebras cannot, while showing that encoder upgrades and directionality provide complementary improvements. The category-specific breakdown and explicit parameter count are strengths that make the empirical comparison falsifiable.","major_comments":[{"comment":"Abstract: the central claim attributes the +29.9pp gain on the five position-shift categories to the addition of directed types, yet the comparison pits a deterministic CKY parser + linear decoder against an undirected AM algebra. No ablation is described that holds the parser architecture, decoder, and training fixed while toggling only the presence of directed types.","section":"Abstract"},{"comment":"Abstract: the claim that 'gains are highly directional' rests on SLOG's category groupings cleanly isolating directional effects, but the manuscript provides no verification (e.g., per-category recursion-depth or lexical statistics) that the groupings are orthogonal to recursion depth or other confounders.","section":"Abstract"},{"comment":"Abstract: concrete numbers with standard deviations and category-specific gains are reported, yet no information is given on how the SLOG categories were constructed, whether any data filtering occurred, or the precise implementation differences between the two symbolic backends beyond the high-level description.","section":"Abstract"}],"minor_comments":[{"comment":"The manuscript would benefit from a table or appendix listing the exact SLOG category definitions and any preprocessing steps applied to the benchmark.","section":null},{"comment":"Clarify the training details (optimizer, learning rate schedule, number of runs) for the reported means and standard deviations to support reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on the abstract and empirical claims. We address each point below and will revise the manuscript to incorporate additional details, verifications, and discussion where feasible.","responses":[{"response":"We agree that a controlled ablation isolating only the presence/absence of directed types (while fixing parser architecture, decoder, and training) is not present and would provide stronger causal evidence. The current comparison contrasts two complete symbolic backends whose core difference is the use of directed CCG types versus an undirected AM algebra; the CKY parser and linear decoder are tailored to the directed representation. An ablation that retrofits directionality into the AM algebra would require substantial new engineering outside the scope of this work. In revision we will add an explicit limitations paragraph acknowledging this gap and noting that the category-specific pattern (gains confined to the five directional phenomena) remains consistent with the role of directionality.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the central claim attributes the +29.9pp gain on the five position-shift categories to the addition of directed types, yet the comparison pits a deterministic CKY parser + linear decoder against an undirected AM algebra. No ablation is described that holds the parser architecture, decoder, and training fixed while toggling only the presence of directed types."},{"response":"The referee is correct that the manuscript does not supply per-category statistics to verify orthogonality. SLOG categories are defined in the original SLOG benchmark paper by the targeted linguistic phenomenon (position shifts for directional effects versus nesting depth for recursive effects). In the revised version we will add a short appendix table reporting, for each SLOG category, mean recursion depth and basic lexical statistics (type-token ratio, average sentence length) to allow readers to assess potential confounding.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that 'gains are highly directional' rests on SLOG's category groupings cleanly isolating directional effects, but the manuscript provides no verification (e.g., per-category recursion-depth or lexical statistics) that the groupings are orthogonal to recursion depth or other confounders."},{"response":"We will expand the experimental section and add an appendix with: (i) a concise summary of SLOG category construction drawn from the SLOG reference, (ii) explicit confirmation that the standard SLOG splits were used with no additional filtering, and (iii) precise implementation details for both backends, including the CKY chart rules for directed CCG types, the single linear decoder architecture and training objective, and exact parameter counts (30K for the CCG decoder versus the AM-Parser configuration).","revision_made":"yes","referee_comment":"[Abstract] Abstract: concrete numbers with standard deviations and category-specific gains are reported, yet no information is given on how the SLOG categories were constructed, whether any data filtering occurred, or the precise implementation differences between the two symbolic backends beyond the high-level description."}],"tokens_in":1442,"tokens_out":640,"duration_ms":29048,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that the paper swaps the symbolic backend for CCG directed types and sees clear gains on the five position-shift categories in SLOG, while AM-Parser stays ahead on the recursive-depth ones. The two improvements look complementary when the encoder is upgraded to DeBERTa.\n\nThe work does a clean job of reporting the category split using SLOG's own groupings and showing that the directional lift is large and consistent across those five categories. Keeping the BERT-base encoder fixed for the head-to-head makes the comparison straightforward, and the standard deviations are reported. The observation that encoder upgrades help the recursive categories more is useful and points to different bottlenecks.\n\nThe soft spot is the missing ablation. The CCG system uses a deterministic CKY parser and a 30k-parameter linear decoder; AM-Parser uses an undirected algebra. Without an experiment that holds the parser, decoder, and training fixed while toggling only the directed types, the +29.9pp cannot be cleanly attributed to directionality rather than the parser architecture. The paper also takes the SLOG category groupings as given without showing they are orthogonal to recursion depth or lexical factors.\n\nThis paper is for people working on semantic parsing and structural generalization. Readers who care about how symbolic direction interacts with neural encoders will find the per-category numbers worth their time. It has enough concrete results on a public benchmark to deserve a serious referee, even though the causal claim about directionality needs tighter controls.\n\nI would send it to review.","headline":"CCG directional types improve SLOG position-shift categories by 30 points over AM-Parser under the same encoder, but the comparison does not isolate directionality from the switch to CKY plus linear decoder.","tokens_in":2390,"tokens_out":390,"would_cite":false,"duration_ms":18154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Directed types in CCG let a parser generalize better to position shifts than an undirected algebra does.","keywords":["CCG","directionality","structural generalization","semantic parsing","SLOG benchmark","AM algebra","position shifts","recursive depth"],"falsifier":"Re-running the comparison after swapping the directionality component alone while keeping every other implementation detail identical, or after re-partitioning SLOG categories to mix position-shift and recursion items, would show whether the reported gap survives.","tokens_in":2622,"feed_emoji":"","tokens_out":685,"duration_ms":18812,"temperature":0.7,"pith_summary":"The paper tests whether adding explicit directionality to the symbolic layer improves structural generalization on the SLOG benchmark. With the same BERT-base encoder, a CCG-based system using directed types reaches 75.9 percent exact match and beats the prior AM-Parser system by five points overall. The advantage appears almost entirely on the five position-shift categories, where the CCG system gains nearly thirty points, while the undirected system stays stronger on recursive-depth categories. Replacing the encoder with DeBERTa-v3-large raises accuracy to 90.7 percent and closes the recursion gap, showing that directionality and encoder capacity address different parts of the generalization problem.","feed_headline":"Directed types lift position-shift accuracy 30 points over undirected algebra","feed_subtitle":"CCG parser beats prior best on SLOG directional tests; larger encoders then close the recursion gap by addressing a separate weakness.","key_machinery":"CCG directed types inside a deterministic CKY parser paired with a single linear decoder that supplies the directional information missing from the AM algebra.","core_discovery":"Redesigning the symbolic backend around CCG directed types (via deterministic CKY plus a single linear decoder) encodes directional distinctions that the AM algebra lacks. Under identical BERT-base encoding this yields higher exact-match accuracy on SLOG, with the improvement concentrated on modifier-position and argument-extraction shifts. The same change leaves recursive-depth performance lower than the AM baseline. Larger encoders then improve the recursive categories more than the directional ones, indicating that directionality moves the remaining bottleneck from the symbolic layer to the neural encoder.","pith_inferences":["A hybrid symbolic layer that combines directed types with recursive-friendly operations might remove both bottlenecks at once.","The same directional redesign could be tested on other semantic parsing benchmarks that contain argument-order or modifier-position variation.","If directionality is the key missing piece, then any undirected algebra will hit a hard ceiling on position-shift generalization regardless of encoder size."],"forward_implications":["Position-shift categories show a 29.9-point gain when directionality is added to the symbolic layer.","Recursive-depth categories remain stronger under the undirected AM algebra than under CCG.","Encoder upgrades produce their largest gains on recursive-depth categories, complementary to the directional gains.","The symbolic layer's 0 percent category ceiling disappears once direction is represented, moving the limit to the neural encoder."],"fun_headline_variants":["CCG directed types add 30 points to position-shift accuracy","Symbolic directionality improves all position-shift categories","Directionality leaves recursive-depth performance lower","Larger encoders close recursion gap after directional change"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The SLOG category groupings separate directional effects from recursion depth and lexical choice, and any performance difference comes from the directed types rather than other differences between the two parsers.","fun_headline_variants_meta":{"raw":{"variants":["CCG directed types add 30 points to position-shift accuracy","Symbolic directionality improves all position-shift categories","Directionality leaves recursive-depth performance lower","Larger encoders close recursion gap after directional change"]},"model":"grok-4.3","cost_usd":0.005282,"raw_usage":{"total_tokens":2475,"prompt_tokens":671,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":52815500,"prompt_tokens_details":{"text_tokens":671,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1748,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":671,"tokens_out":56,"duration_ms":13764,"temperature":1.0,"reasoning_tokens":1748,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-03T14:21:09.005340+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-running the comparison after swapping the directionality component alone while keeping every other implementation detail identical, or after re-partitioning SLOG categories to mix position-shift and recursion items, would show whether the reported gap survives.","supporting_citations":[],"review_version":1}