Pith. sign in

REVIEW 3 major objections 6 minor 59 references

STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read STARE's two-stage structure-aware retriever plus syntactic plug-in consistently improves in-context semantic parsing across four benchmarks.

desk verdict A plausible and modular structure-aware retriever plus a probe-injection plug-in, but the claimed 'consistent' gains are partly inflated by tuning MLI on the Spider evaluation set; still worth serious refereeing. read the letter →

arxiv 2508.20944 v1 pith:Z3BWM3EH submitted 2025-08-28 cs.CL

Jiaqian Li , Qisheng Hu , Jing Li , Wenya Wang This is my paper
classification cs.CL
keywords in-contextlearningexemplarselectionsemanticparsingstructuralalignmenttreeeditdistancehidden-stateinterventionlinguisticprobingretrieval-augmentedprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STARE is a two-stage exemplar-selection method for in-context learning on semantic parsing. It claims that choosing exemplars that are both semantically close and structurally aligned to the test query beats both embedding-similarity retrievers and expensive proxy-task retrievers that query an LLM to build training labels. The first stage builds structure-aware contrastive pairs from parse trees; the second stage, Middle-Layer Injection (MLI), amplifies syntactic information in the retriever's hidden states using a direction extracted from linguistic probes. Across MTop, SMCalFlow, TreeDST, and Spider, the paper reports consistent gains over baselines under several inference LLMs, with MLI contributing average gains of about 2.2 exact-match points on the first three tasks and 0.7 execution / 3.3 exact-match points on Spider. If correct, this offers a model-agnostic, low-cost way to improve few-shot parsing without querying a large language model during training.

What carries the argument

The paper's central object is the structure-aware exemplar retriever plus its Middle-Layer Injection (MLI) plug-in. The retriever is trained with contrastive pairs whose positives are chosen by tree-edit-distance similarity inside MinHash/LSH semantic buckets, so the embedding space is organized by both meaning and compositional form. MLI is the load-bearing intervention: it trains linear probes for part-of-speech, dependency, and phrase-type labels, takes the first right singular vector of each probe's weight matrix as a syntactic direction u_prop, and injects λu_prop into hidden representations at layer 8. This single vector addition is what steers retrieval toward structurally aligned exe

What would settle it

Decisive checks: (1) freeze the MLI configuration chosen on one dataset's development set and apply it unchanged to other tasks and to a held-out Spider test split; if the reported gains disappear or reverse, the improvements come from per-task development-set tuning rather than from the syntactic direction itself. (2) Inject a random unit vector at layer 8 with the same intensity λ; if random-direction injection performs as well as the probe-derived u_prop, the probe is not doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that structural alignment between a test query and candidate exemplars is a distinct, exploitable signal for in-context semantic parsing, and that it can be captured without proxy-model feedback. STARE first buckets parses by semantic similarity using MinHash/LSH hashing of normalized tokens and keywords, then, within each bucket, constructs contrastive pairs by normalized Zhang–Shasha tree edit distance: the most structurally similar parse is the positive, the least similar are hard negatives. This training produces a BERT retriever whose cosine similarity ranks exemplars by both meaning and compositional form. The second component, Middle-Layer Injection (MLI),

Load-bearing premise

The method assumes that a syntactic direction learned from a generic English treebank, added to a fixed middle layer of the retriever, improves exemplar selection for each semantic-parsing task, and that the property and intensity chosen on a development set keep working on test queries.

Editorial extensions

If this is right

  • If STARE is right, exemplar selection for semantic parsing can be improved without querying a proxy LLM during training, removing a major cost and a source of overfitting for methods like EPR and CEIL.
  • The MLI plug-in should transfer across retrievers: the paper shows it improves a plain BERT retriever and EPR as well as STARE itself, so it can be bolted onto existing pipelines.
  • The best linguistic property to inject is task-dependent—lexical POS cues help Spider, while dependency and phrase-type cues help SMCalFlow and TreeDST—so property choice can be treated as a cheap per-task hyperparameter.
  • Because MLI only modifies the retriever, it composes with any frozen inference LLM; the paper verifies this across multiple models and prompt formats.
  • MinHash/LSH semantic bucketing plus sparse tree-edit-distance comparisons keeps structure-aware retrieval scalable, avoiding exhaustive pairwise TED computations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer the same recipe could be pointed at other structured-output tasks—code generation, JSON or knowledge-graph construction, tool-use—where lexical similarity under-specifies the output skeleton; the probe directions are language-level, not task-level, so no task-specific probe training would be needed.
  • We infer that the layer-8 sweet spot is not a constant of nature: for encoders with different depth or initialization, the injection layer should be re-anchored to the probe's syntactic peak rather than fixed.
  • We infer that a cleaner transfer test would freeze one MLI configuration (property, intensity, layer) and apply it unchanged across tasks; the paper tunes these per task, which makes the reported gains an upper bound on what a fully hands-off deployment would deliver.
  • We infer that structural positivity could be expanded beyond tree-edit distance—for example, paraphrases of the same query that preserve the parse skeleton would give contrastive pairs that isolate structure from surface wording.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes STARE, a two-stage exemplar-selection method for in-context learning in semantic parsing. Stage 1 fine-tunes a BERT retriever with contrastive pairs mined from gold parses, combining MinHash/LSH semantic bucketing and normalized tree-edit-distance structural filtering. Stage 2, Middle-Layer Injection (MLI), trains linear probes on external treebanks (UD, PTB), extracts a dominant direction via SVD, and injects it into layer 8 of the retriever's hidden states as h' = h + λu_prop. Experiments on MTop, SMCalFlow, TreeDST, and Spider with several inference LLMs report improvements over EPR, CEIL, MLSM, Skill-KNN, and similarity/diversity baselines, and also show MLI as a plug-in for BERT and EPR retrievers. The paper includes three-seed averaging and a planned code release.

Significance. If the empirical claims hold, STARE would be a useful, low-cost alternative to proxy-LLM-based exemplar retrieval for structured prediction: it avoids repeated LLM queries during training, uses gold parse structure directly, and the MLI plug-in is conceptually simple and model-agnostic. The use of external treebanks for probing, the modular integration with existing retrievers, and the breadth of inference models are concrete strengths. However, the current support is materially weakened by the Spider evaluation protocol, where MLI hyperparameters are selected on the same development set used to report final numbers, and by the absence of variance or significance information. The headline claim of consistent superiority is also contradicted by an admitted loss on SMCalFlow with DeepSeek-V3. The central idea is defensible, but the evidence as presented needs substantial strengthening before the claimed cross-benchmark consistency can be accepted.

major comments (3)
  1. [Section 6 / Appendix A / Table 11] Spider is evaluated on the development set because the test set is not public (Appendix A, Table 5), and Section 6 states that the MLI configuration is chosen by sweeping λ and the linguistic property on the development set, then applying the best configuration at test time. For Spider, these two sets coincide. Therefore the Spider results in Tables 2 and 4 (e.g., STARE +3.3 EM, EPR +8.7 EM) are maxima over the tuning grid on the evaluation instances rather than clean transfer results. This is selection-on-test leakage: it inflates both absolute scores and MLI's apparent contribution, and it violates the paper's stated transfer premise. Please re-evaluate Spider using a held-out split that was not used for selecting MLI hyperparameters, or obtain an unseen test set, and report the configurations chosen on that separate split.
  2. [Abstract / Section 5.1, Table 1] The Abstract and Conclusion claim that STARE 'consistently outperforms existing baselines,' but Table 1 shows that on SMCalFlow with DeepSeek-V3, STARE obtains 61.9 EM versus 63.7 for CEIL and 63.5 for EPR. Section 5.1 acknowledges this only as 'except on SMCalFlow under DeepSeek-V3,' and the subsequent average-gain sentence obscures the pattern. The claimed consistency is therefore overstated. Please revise the Abstract and Conclusion to state the exception explicitly, and report per-task win/loss counts or a complete comparison matrix rather than only averaged gains.
  3. [Appendix E / Tables 1–4] Appendix E says each experiment is run with three random seeds and averaged, but the main tables report only point estimates. Many reported advantages over the strongest baseline are 1–3 EM points (e.g., MTop with Llama3: STARE 69.5 vs CEIL 68.3; TreeDST with 4o-mini: STARE 45.5 ties EPR 45.5), and there is one negative cell. Without standard deviations, per-seed values, or paired significance tests, the claim of consistent improvement is not statistically supported. Please include variance estimates and, where possible, significance tests, particularly for the MTop/SMCalFlow/TreeDST results that are the cleanest transfer evidence.
minor comments (6)
  1. [Section 5.1] The phrase '1.35% gain' should be '1.35 percentage points' or should clearly define the metric and the baseline over which the average is taken.
  2. [Figure 4] The y-axis label 'ICL performance' does not specify the metric; please label it as Exact Match (or Execution accuracy, as appropriate) so the tuning plots are interpretable.
  3. [Section 3.2.1 / Eq. (3)] The LSH similarity threshold τ is a free parameter that controls the candidate pool, but no sensitivity analysis is provided for τ, and its chosen value is not reported in Appendix I. A brief analysis or at least a statement of the chosen value would help reproducibility.
  4. [Section 3.3 / Eq. (5)] The method text says the injection layer is a hyperparameter, while Section 3.3 and Appendix I state that all injections are applied at Layer 8 and Section 6 pre-selects Layers 4/8/12. Please clarify whether Layer 8 was chosen a priori or through the development sweep, and whether any other layers were competitive in the final configuration.
  5. [Limitations] The Limitations section appropriately acknowledges the linear-injection simplification and the lack of reasoning-method comparisons, but it does not acknowledge the Spider development-set tuning issue discussed above. This protocol limitation should be disclosed in the Limitations or Experimental Setup section.
  6. [Appendix B / Related Work] The naming of TST is used both for 'Target Similarity Tuning' and as the method from Poesia et al. (2022), while the related work section cites the same work under 'Synchromesh'. Please make the naming consistent and clarify which component is adopted from that work.

Circularity Check

1 steps flagged · score 6.0 of 10

Spider MLI gains reduce to dev-set tuning: reported improvements are the maximum over the hyperparameter grid on the evaluation set.

  1. fitted input called prediction [Section 6 (MLI Configurations), Appendix A (Spider), Tables 4 and 11]
    "To determine the most effective linguistic property for injection and the optimal intensity λ, controlled experiments are conducted on the development set. ... We select the configuration that achieves the highest development performance for each task and apply it in test-time evaluation. ... Following standard practice, we evaluate on the development set since the test set is not publicly released. ... MLI configurations (property - intensity) for each inference model across datasets."

    For Spider, the development set is the evaluation set: Appendix A states that Spider is evaluated on the dev set because the test set is not public. Section 6 then says the MLI property and intensity λ are chosen as the configuration with the highest development performance. Therefore the reported Spider gains (e.g., EPR+MLI EM +8.7 in Table 4, with configuration POS-5 in Table 11) are the argmax of the tuning grid on the same 1,034 queries used to compute the final numbers. The 'MLI gain' on Spider is forced by the selection rule itself—it is the value of the dev-set objective at its optimum—rather than an out-of-sample measurement. This is a fitted parameter (λ and property) evaluated on its own fitting set, so the Spider portion of the claim that MLI gives an average gain of 0.7 EX / 3.

full rationale

The paper's underlying derivation is not circular: the structure-aware retriever is trained with contrastive pairs built from MinHash semantic bucketing and normalized tree edit distance on gold parses (Eqs. 3–4), and MLI extracts directions via SVD from probes trained on external UD/Penn Treebank data, then injects h' = h + λuprop (Eq. 5). None of these steps uses the downstream ICL scores as supervision. The MLSM baseline is a self-citation (Liu et al. 2024 shares author Wenya Wang), but it appears only as a comparison method and is not load-bearing. The significant circularity is confined to the Spider evaluation protocol: the same dev set is used both to select the MLI configuration and to report the final result, so the reported Spider gains are maxima over the tuning grid rather than test-time transfer. This does not invalidate the three clean-split benchmarks (MTop, SMCalFlow, TreeDST), which have separate test sets, but it weakens the claim that STARE/MLI consistently outperforms baselines across all four benchmarks. If the Spider dev-as-test leakage were removed, the remaining evidence would support a low circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central results rest on a handful of fitted hyperparameters (λ, property, layer, LSH threshold) and on the assumption that linear probe directions, TED similarity, and hashed Jaccard semantics are valid proxies for exemplar utility. The probe direction u_prop is an extracted vector from existing representations, not an invented entity, so no independent evidence is needed in that sense.

free parameters (4)
  • MLI injection intensity λ = 0.5 to 6 depending on task/model (Table 11)
    Selected by sweeping λ over a range on the development set (Section 6). The gains credited to MLI depend on this per-task, per-model choice, and for Spider the development set equals the test set.
  • MLI linguistic property = POS, DEPS, or PT (Table 11)
    The augmented syntactic property is chosen per dataset and inference model to maximize dev accuracy; different choices change results by several EM points.
  • MLI injection layer N = 8 for all reported configurations
    Layers 4, 8, and 12 were pre-selected, and layer 8 was chosen based on dev-set ablations (Section 6, Appendix I).
  • LSH similarity threshold τ = not reported
    Eq. 3 defines the candidate pool using Jaccard similarity ≥ τ, but no value or sensitivity analysis is given, so the contribution of this free threshold is unquantified.
assumptions (5)
  • domain assumption Adding a linear probe direction amplifies syntactic information in hidden states without breaking the retriever (h' = h + λu_prop).
    Central mechanism of MLI (Eq. 5). The paper's Limitations section concedes this assumes linear, independent linguistic directions.
  • domain assumption Normalized Zhang-Shasha tree edit distance with unit edit costs on constructed parse trees captures the structural alignment relevant to exemplar utility.
    Used to define positives and hard negatives in retriever training (Eq. 4). No evidence is given that unit-cost TED tracks ICL performance.
  • domain assumption Jaccard similarity over discrete parse features, approximated by MinHash/LSH, is a good proxy for semantic relevance.
    Semantic bucketing in Section 3.2.1 assumes this; the threshold τ is not reported.
  • domain assumption Linear probes trained on UD Treebank and Penn Treebank transfer to BERT hidden representations of in-domain semantic-parsing utterances.
    MLI probes are trained on external treebanks and applied to task-specific queries (Sections 3.3 and 4.3).
  • domain assumption Mean pooling of BERT final-layer hidden states yields a retrieval representation adequate for contrastive learning.
    Section 3.2.3 states sentence-level representations are obtained via mean pooling; no comparison to CLS or pooling alternatives is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment." pith.science (2026). https://pith.science/paper/Z3BWM3EH

@misc{pith2026250820944,
  author       = {Pith},
  title        = {Pith review of: STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3BWM3EH}},
  note         = {Machine review of arXiv:2508.20944}
}
read the original abstract

In-Context Learning (ICL) has become a powerful paradigm that enables LLMs to perform a wide range of tasks without task-specific fine-tuning. However, the effectiveness of ICL heavily depends on the quality of exemplar selection. In particular, for structured prediction tasks such as semantic parsing, existing ICL selection strategies often overlook structural alignment, leading to suboptimal performance and poor generalization. To address this issue, we propose a novel two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability, and performance. First, we fine-tune a BERT-based retriever using structure-aware supervision, guiding it to select exemplars that are both semantically relevant and structurally aligned. Then, we enhance the retriever with a plug-in module, which amplifies syntactically meaningful information in the hidden representations. This plug-in is model-agnostic, requires minimal overhead, and can be seamlessly integrated into existing pipelines. Experiments on four benchmarks spanning three semantic parsing tasks demonstrate that our method consistently outperforms existing baselines with multiple recent LLMs as inference-time models.

Figures

Figures reproduced from arXiv: 2508.20944 by the authors.

Figure 1
Figure 1. Comparison between semantic-only and structure-aware similarity based one-shot prompting with Llama-3-8B. Exemplar selection methods can be broadly cate￾gorized into two aspects: proxy-task-based (Rubin et al., 2022; Shi et al., 2022; Li et al., 2023; Ye et al., 2023) and similarity-based approaches (Das et al., 2021; Hu et al., 2022; An et al., 2023). Proxy-task￾based methods extensively query a proxy LLM to evalua… view at source ↗
Figure 2
Figure 2. Overview of our proposed framework STARE. The backbone retriever is trained via contrastive learning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of Middle-Layer Injection (MLI). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: ICL performance on the dev sets of MTop, SMCalFlow, and TreeDST (reported as Exact Match accuracy), [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Effect of applying MLI at different injection [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 35 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. https://arxiv.org/abs/1608.04207 Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

    arXiv 2017

  4. [4]

    Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. https://aclanthology.org/2023.findings-acl.564/ In-context examples selection for machine translation . In Findings of the Association for Computational Linguistics: ACL 2023

    work page 2023

  5. [5]

    Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. 2023. https://aclanthology.org/2023.emnlp-main.831/ Skill-based few-shot selection for in-context learning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  6. [6]

    Crawford

    Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, and Jean et al. Crawford. 2020. https://aclanthology.org/2020.tacl-1.36/ Task-oriented dialogue as dataflow synthesis . Transactions of the Association for Computational Linguistics, 8

    work page 2020

  7. [7]

    Andrei Z. Broder. 1997. https://doi.org/10.1109/SEQUEN.1997.666900 On the resemblance and containment of documents . In Proceedings of the Compression and Complexity of Sequences 1997, pages 21--29. IEEE Computer Society

    arXiv 1997

  8. [8]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. https://arxiv.org/abs/2005.14165 Language models are few-shot learners . arXiv preprint arXiv:2005.14165

    arXiv 2020

Show all 59 references
  1. [9]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2024. https://arxiv.org/abs/2212.03827 Discovering latent knowledge in language models without supervision . Preprint, arXiv:2212.03827

    2024 arXiv

  2. [10]

    Williams, Hong Yu, Diarmuid O Seaghdha, and Anders Johannsen

    Jianpeng Cheng, Devang Agrawal, Hector Martinez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Shaona Ghosh, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid O Seaghdha, and Anders Johannsen. 2021. https://arxiv.org/a...

    2021 arXiv

  3. [11]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \"i c Barrault, and Marco Baroni. 2018. https://aclanthology.org/P18-1198/ What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Meeti...

    2018

  4. [12]

    Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. https://aclanthology.org/2021.emnlp-main.755/ Case-based reasoning for natural language queries over knowledge bases . In Proceedings o...

    2021

  5. [13]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. https://arxiv.org/abs/24...

    2025 arXiv

  6. [14]

    Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. https://aclanthology.org/2021.tacl-1.10/ Amnesic probing: Behavioral explanation with amnesic counterfactuals . Transactions of the Association for Computational Linguistics, 9

    2021

  7. [15]

    Kawin Ethayarajh. 2019. https://aclanthology.org/D19-1006/ How contextual are contextualized word representations? C omparing the geometry of BERT , ELM o, and GPT -2 embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the...

    2019

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  9. [17]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  10. [18]

    Smith, and Mari Ostendorf

    Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. https://aclanthology.org/2022.findings-emnlp.193/ In-context learning for few-shot dialogue state tracking . In Findings of the Association for Computational Linguistics: EMNLP 2022

    2022

  11. [19]

    Ganesh Jawahar, Beno \^i t Sagot, and Djam \'e Seddah. 2019. https://aclanthology.org/P19-1356/ What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  12. [20]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, and Ethan Perez et al. 2022. https://arxiv.org/abs/2207.05221 Language models (mostly) know what they know . Preprint, arXiv:2207.05221

    2022 arXiv

  13. [21]

    Jimin Lee, Ingeol Baek, Byeongjeong Kim, and Hwanhee Lee. 2025. https://arxiv.org/abs/2502.11438 Safe-sql: Self-augmented in-context learning with fine-grained example selection for text-to-sql . Preprint, arXiv:2502.11438

    2025

  14. [22]

    Tal Levy, Omer Goldman, and Reut Tsarfaty. 2023. https://arxiv.org/abs/2310.15905 Is probing all you need? indicator tasks as an alternative to probing embedding spaces . Preprint, arXiv:2310.15905

    2023 arXiv

  15. [23]

    Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. https://aclanthology.org/2021.eacl-main.257/ MTOP : A comprehensive multilingual task-oriented semantic parsing benchmark . In Proceedings of the 16th Conference of the European Chapter...

    2021

  16. [24]

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. https://arxiv.org/abs/2306.03341 Inference-time intervention: Eliciting truthful answers from a language model . Preprint, arXiv:2306.03341

    2024 arXiv

  17. [25]

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. https://aclanthology.org/2023.acl-long.256/ Unified demonstration retriever for in-context learning . In Proceedings of the 61st Annual Meeting of the Association fo...

    2023

  18. [26]

    Xiaonan Li and Xipeng Qiu. 2023. https://aclanthology.org/2023.findings-emnlp.411/ Finding support examples for in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023

    2023

  19. [27]

    Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, and Ruixiang Tang. 2025 a . https://arxiv.org/abs/2504.04633 M ^2 iv: Towards efficient and fine-grained multimodal in-context learning via representation engineering . Preprint, arXiv:2504.04633

    2025 arXiv

  20. [28]

    Yanshu Li, Tian Yun, Jianjiang Yang, Pinyuan Feng, Jinfa Huang, and Ruixiang Tang. 2025 b . Taco: Enhancing multimodal in-context learning via task mapping-guided sequence configuration. arXiv preprint arXiv:2505.17098

    2025

  21. [29]

    Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. https://arxiv.org/abs/2012.12627 Bridging textual and tabular data for cross-domain text-to-sql semantic parsing

    2020 arXiv

  22. [30]

    Hui Liu, Wenya Wang, Hao Sun, Chris Xing Tian, Chenqi Kong, Xin Dong, and Haoliang Li. 2024. https://arxiv.org/abs/2406.11890 Unraveling the mechanics of learning-based demonstration selection for in-context learning . Preprint, arXiv:2406.11890

    2024 arXiv

  23. [31]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://aclanthology.org/2022.deelio-1.10/ What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022)

    2022

  24. [32]

    Liu, Matt Gardner, Yonatan Belinkov, Matthew E

    Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. https://aclanthology.org/N19-1112/ Linguistic knowledge and transferability of contextual representations . In Proceedings of the 2019 Conference of the North American Chapter of the Ass...

    2019

  25. [33]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. https://aclanthology.org/J93-2004/ Building a large annotated corpus of english: The penn treebank . Computational Linguistics, 19(2):313--330

    1993

  26. [34]

    Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, and Slav Petrov. 2013. https://aclanthology.org/L16-1262/ Universal dependencies: A multilingual treebank collection . In Proceedings of the 10th International Conf...

    2013

  27. [35]

    Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. https://aclanthology.org/2023.findings-emnlp.996/ Enhancing text-to- SQL capabilities of large language models: A study on prompt design strategies . In Findings...

    2023

  28. [36]

    OpenAI . 2023. Gpt-3.5 turbo. https://platform.openai.com/models/gpt-3.5-turbo. Accessed: 2025-05-12

    2023

  29. [37]

    OpenAI . 2024. Gpt-4o-mini. https://platform.openai.com/models/gpt-4o-mini. Accessed: 2025-05-12

    2024

  30. [38]

    Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. https://aclanthology.org/2021.emnlp-main.607/ Controllable semantic parsing via retrieval augmentation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    2021

  31. [39]

    Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. 2021. https://aclanthology.org/2021.acl-long.284/ Value-agnostic conversational semantic parsing . In Pro...

    2021

  32. [40]

    Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. https://arxiv.org/abs/2201.11227 Synchromesh: Reliable code generation from pre-trained language models . Preprint, arXiv:2201.11227

    2022 arXiv

  33. [41]

    Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. https://aclanthology.org/2022.emnlp-main.211/ RASAT : Integrating relational structures into pretrained S eq2 S eq model for text-to- SQL . In Proceed...

    2022

  34. [42]

    Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. https://aclanthology.org/2021.conll-1.15/ Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction . In Proceedings of the 25th Conference on Computatio...

    2021

  35. [43]

    Stephen Robertson and Hugo Zaragoza. 2009. Foundations and Trends in Information Retrieval, https://doi.org/10.1561/1500000019 3(4):333--389

    2009 doi

  36. [44]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://aclanthology.org/2022.naacl-main.191/ Learning to retrieve prompts for in-context learning . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2022

  37. [45]

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. https://aclanthology.org/2021.emnlp-main.779/ PICARD : Parsing incrementally for constrained auto-regressive decoding from language models . In Proceedings of the 2021 Conference on Empirical Methods in Natural Lang...

    2021

  38. [46]

    Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2022. https://aclanthology.org/2022.findings-emnlp.384/ XRICL : Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to- SQL semantic parsing . In Findings of the Association for Computational Linguistics: EMNLP 2022

    2022

  39. [47]

    Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. https://aclanthology.org/2021.emnlp-main.608/ Constrained language models yield few-shot semantic parsers . In...

    2021

  40. [48]

    Richard Shin and Benjamin Van Durme. 2022. https://aclanthology.org/2022.naacl-main.396/ Few-shot semantic parsing with language models trained on code . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2022

  41. [49]

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. https://aclanthology.org/2022.findings-acl.48/ Extracting latent steering vectors from pretrained language models . In Findings of the Association for Computational Linguistics: ACL 2022

    2022

  42. [50]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. https://arxiv.org/abs/1905.05950 Bert rediscovers the classical nlp pipeline

    2019 arXiv

  43. [51]

    Mycal Tucker, Peng Qian, and Roger Levy. 2021. https://doi.org/10.18653/v1/2021.findings-acl.76 What if this modified that? syntactic interventions with counterfactual embeddings . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862--875, O...

    2021 doi

  44. [52]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. https://arxiv.org/abs/1807.03748 Representation learning with contrastive predictive coding . Preprint, arXiv:1807.03748

    2019 arXiv

  45. [53]

    Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2021. https://arxiv.org/abs/2106.02902 Bertnesia: Investigating the capture and forgetting of knowledge in bert . Preprint, arXiv:2106.02902

    2021 arXiv

  46. [54]

    Chenguang Wang, Xiao Liu, and Dawn Song. 2020. https://arxiv.org/abs/2010.11967 Language models are open knowledge graphs . Preprint, arXiv:2010.11967

    2020 arXiv

  47. [55]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. https://arxiv.org/abs/2212.10375 Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering . Preprint, arXiv:2212.10375

    2023 arXiv

  48. [56]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. https://arxiv.org/abs/2302.05698 Compositional exemplars for in-context learning

    2023 arXiv

  49. [57]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. https://aclanthology.org/D18-1425/ S pider: A large-scale human-labeled dataset for complex and cross-domain semanti...

    2018

  50. [58]

    Zelle and Raymond J

    John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96)

    1996

  51. [59]

    Zhang and D

    K. Zhang and D. Shasha. 1989. https://doi.org/10.1137/0218082 Simple fast algorithms for the editing distance between trees and related problems . SIAM Journal on Computing, 18(6):1245--1262

    1989 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.