Pith. sign in

REVIEW 4 major objections 5 minor 42 references

PBE Meets LLM: When Few Examples Aren't Few-Shot Enough

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows GPT-4o beats classic data-wrangling tools on example-driven table transformations, with a Foofah-first hybrid reaching the top score.

desk verdict The headline accuracy gap is confounded: GPT-4o sees the test inputs at generation time, so the comparison with Foofah/Prose is not apples-to-apples. read the letter →

arxiv 2507.05403 v1 pith:MA45XDRX submitted 2025-07-07 cs.DB

Shuning Zhang , Yongjoo Park This is my paper · ORCID
classification cs.DB
keywords programmingbyexamplelargelanguagemodelstabulardatatransformationfew-shotpromptingmulti-tryverificationhybridprogramsynthesiswranglingGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can do Programming by Example (PBE) for tabular data: given one example input table and its output table, the model must write a function that transforms unseen tables in the same way. On the Foofah and Prose benchmarks, a single-prompt GPT-4o reaches 0.797 weighted accuracy, against 0.571 for Foofah and 0.473 for Prose, and it keeps relatively even accuracy across both benchmark formats while each classic system drops sharply on the other's dataset. The authors also find that giving the model a short list of allowed operations helps some multi-try configurations, and that a hybrid pipeline running Foofah first and falling back to GPT-4o reaches 0.863. The practical stake is that users could specify data-wrangling intent by example and get working code without writing a program; the paper also identifies the remaining weak spots, namely ambiguous examples and tasks requiring commonsense reasoning.

What carries the argument

The carrying mechanism is the example-pair prompt: each task feeds GPT-4o an input table and an output table in JSON and asks for a Python function, which is then executed on held-out inputs and scored by row-level exact match. Around that core, the paper adds two refinements. A multi-try verification loop runs the generated code on the example input, sends any mismatch to a second GPT-4o pass that identifies structural errors, and feeds that list of fixes back to the original generator; this loop is what lifts the best knowledge-augmented configuration to 0.827. A hybrid wrapper runs the Foofah solver first and calls GPT-4o only on the cases Foofah cannot handle, which reaches 0.863 overall. These components carry the argument because each accuracy number in the study is produced by running one of these pipelines once on every benchmark case.

What would settle it

Run the same one-shot prompt on the Foofah and Prose scenarios after replacing values, row order, and column names with novel equivalents; if accuracy falls well below the reported 0.797, the headline gap is largely memorization. Repeating each configuration several times would also show whether the 0.766-to-0.863 spread between methods is within run-to-run noise.

Watch

Extended reading notes

Core claim

The central claim is that modern LLMs, without fine-tuning or any change to the model architecture, solve tabular data transformation tasks specified only by input-output examples more accurately and across more diverse input formats than conventional search-based PBE systems. On the combined Foofah-Prose evaluation, one-shot GPT-4o achieves 0.797 weighted exact-match accuracy against 0.571 for Foofah and 0.473 for Prose; Foofah reaches 1.0 on its own Proactive subset but only 0.300 on Prose's dataset, while Prose reaches 0.949 on its own dataset and drops sharply on Foofah's, whereas the LLM stays relatively flat across the subsets. The paper further claims that providing Foofah's operation vocabulary as a prompt constraint improves the best multi-try result to 0.827 and that a Foofah-first, GPT-4o-fallback hybrid reaches 0.863, the highest score observed. The failure analysis claims that LLMs still fail systematically when the example invites a wrong structural inference, such as reading a column as an index, and when the transformation needs real-world knowledge, such as interpreting a two-digit year as a past date.

Load-bearing premise

The comparison assumes that GPT-4o has not memorized the public Foofah and Prose benchmark answers, so its higher accuracy counts as transformation reasoning rather than memory retrieval.

Editorial extensions

If this is right

  • A non-expert can specify a table transformation by example and receive a working Python function from an LLM at accuracy comparable to or better than dedicated PBE tools.
  • LLMs remove the input-format specialization of classic PBE systems: the same model covers Foofah-style relational tables and Prose-style semi-structured tables without retraining.
  • A small prompt-level operation vocabulary is a cheap substitute for fine-tuning and improves the best multi-try configuration from 0.786 to 0.827.
  • Ambiguity remains the main failure surface: in the class of tasks shown in Figure 4, the model's wrong structural inference caused 87% of test cases to fail.
  • A hybrid that lets a deterministic solver go first and uses the LLM as a fallback attains the highest observed accuracy, 0.863, so the two families of methods are complementary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper runs each configuration once, the ordering between close results such as one-shot at 0.797 versus multi-tries at 0.786 should be treated as provisional until repeated sampling gives error bars.
  • The paper does not test for benchmark leakage; re-prompting GPT-4o on the same transformations with changed values, row order, and column names is the cheapest experiment that would separate reasoning from memorization.
  • If the accuracy ordering is stable, the Foofah-first hybrid rule could generalize to any cheap deterministic PBE solver, reserving the LLM for residual hard cases and lowering cost relative to an LLM-only pipeline.
  • The same example-pair prompting plus verification recipe could transfer to adjacent under-specified tabular tasks, such as data cleaning, schema mapping, and query synthesis, where examples already serve as the specification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper evaluates LLMs on Programming-by-Example (PBE) tasks for tabular data transformation. The authors benchmark GPT-4o and other LLMs on the Foofah and Prose datasets, varying prompt strategy (one-shot vs. multi-tries), external knowledge (none vs. Foofah-inspired operation set), and a hybrid that falls back to GPT-4o when Foofah fails. The headline results are that GPT-4o one-shot achieves 0.797 weighted accuracy versus 0.571 for Foofah and 0.473 for Prose, and that the Foofah+GPT-4o hybrid reaches 0.863. The paper also analyzes failure cases caused by ambiguity and multi-dimensional reasoning.

Significance. If the results were sound, the paper would provide a useful comparison of LLM-based PBE against classical synthesis, and the failure analysis would help the community understand LLM limitations. The artifact release (GitHub) and the cross-domain evaluation are strengths. However, the validity of the head-to-head comparison is undermined by the test-input leakage in the prompt design, the lack of repeated runs, and inconsistencies in the reported numbers. The contribution is therefore more of a case study than a rigorous evaluation in its current form.

major comments (4)
  1. [Section 3.2.2] The 'No Knowledge base prompt' includes the test input list in the prompt, while Section 3.4 shows that Foofah and Prose only receive the example rows. This gives GPT-4o access to the exact inputs on which the generated function is later evaluated, contradicting the abstract's claim that functions are 'tested on unseen inputs'. The LLM can condition its program on the specific test rows, for instance by hard-coding outputs or exploiting structural patterns in the test set, so the reported accuracy gap over Foofah/Prose is not a fair measure of generalization. The experiment should be re-run with test inputs omitted from the prompt, or the claim should be narrowed to in-sample transformation accuracy.
  2. [Section 4] Each approach is executed once per task, with no replication, confidence intervals, or statistical tests. The paper acknowledges this in its preamble but then draws quantitative conclusions, including that one-shot (0.797) is 'slightly better' than multi-tries (0.786), that knowledge improves multi-tries from 0.786 to 0.827, and that the hybrid reaches 0.863. Given that these differences are of comparable size to the likely run-to-run variability of LLM code generation, the paper overstates the reliability of the ordering. At minimum, repeated runs (e.g., 5–10) with variance reporting are needed for the central comparison.
  3. [Table 1 vs. Section 3.1] The row labeled 'Gemini-3.5' reports 0.714, but Section 3.1 does not mention a Gemini-3.5 model; it reports 'Claude 3.5 (71.41%)'. There is no public model named Gemini-3.5. The label is almost certainly a typo, yet it appears in the main results table and obscures which model actually achieved 71.4% accuracy. This must be corrected and the affected interpretation clarified.
  4. [Table 2] The numbers are not reconcilable with Table 1. For the Prose dataset, Table 1 reports Foofah accuracy of 0.300 and Foofah+GPT-4o Multi-tries accuracy of 0.883. Table 2 states that 78.95% of Prose test cases are passed to the LLM and the LLM solves 45.19% of the passed cases. If 'passed' means Foofah failed and 'solved' means the LLM produced the correct output, the hybrid accuracy would be approximately 0.2105 + 0.7895 × 0.4519 ≈ 0.567, far below the 0.883 in Table 1. The definitions of 'passed to LLM' and 'solved by LLM' need to be stated precisely, and the two tables need to be made consistent.
minor comments (5)
  1. [Section 1] 'rerely' should be 'rarely' in the sentence 'we evaluate LLMs on a rerely explored area PBE tasks'.
  2. [Section 2.1.3] 'Foofha proposed' is a typo for 'Foofah proposed'; the paper also alternates between 'Foofah' and 'FooFah' throughout, which should be standardized.
  3. [Section 3.4] The text first says 'Exact match accuracy was used for evaluation' but then describes row-level partial credit; rename the metric to avoid confusion.
  4. [Table 1 caption] The dataset abbreviations (ProgEx, Wrgler, Potter, Proact) are not defined; the caption should expand them to their full benchmark names.
  5. [Section 2.1.2] The citation [9] is a lecture, not the original Prose system paper; the meaning of the 'Prose' baseline is ambiguous between the Microsoft Prose framework and the 'Prose' dataset column in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an empirical benchmark comparison with no fitted parameters, self-citation chains, or definitional reductions.

full rationale

This paper is an empirical evaluation, not a derivation, so the circularity patterns mostly do not apply. The headline accuracy numbers (GPT-4o one-shot 0.797 vs Foofah 0.571 and Prose 0.473, hybrid 0.863) are measured on the public Foofah/Prose benchmarks rather than fitted from those benchmarks; no parameter is fitted to a subset and then renamed as a prediction. The 'extra knowledge' prompt reuses Foofah's operation set (Sec. 3.2.2), but as a disclosed prompt ingredient, not as a hidden derivation. The hybrid (Sec. 3.3) is a defined fallback algorithm; its gain is a measured ensemble property, not an equation that reduces to its inputs. There are no load-bearing self-citations or imported uniqueness theorems. One non-circular validity concern: Sec. 3.2.2 states the test input list was included in the prompt while the abstract claims functions are tested on 'unseen inputs'; this is an evaluation confound (the LLM sees unlabeled test inputs at generation time), but it is not a circular derivation. A leakage check would strengthen the paper, but its absence does not make the reasoning circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The paper is an empirical evaluation; its reliability depends on dataset validity, faithful baseline execution, metric choice, and sampling stability.

assumptions (4)
  • domain assumption The ground-truth transformations and the split into example and test are correct as provided by the Foofah and Prose datasets.
    Section 3.4 uses these datasets without re-validation; if any ground-truth labels are wrong, all accuracy numbers move.
  • domain assumption Foofah and Prose baselines are executed in their intended configuration.
    Section 3.4 lists them as baselines but does not describe how they were invoked; incorrect wrapping would bias the comparison.
  • domain assumption Weighted average accuracy across different transformation types is the appropriate aggregation.
    Section 3.4 defines weighted average to avoid over-weighting rare transformations; if weights are arbitrary, the headline 0.863 is not comparable across methods.
  • domain assumption Single invocation per model gives a stable estimate of accuracy.
    Section 4 says each approach was run once; if the LLM's sampling is stochastic, this ignores variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PBE Meets LLM: When Few Examples Aren't Few-Shot Enough." pith.science (2026). https://pith.science/paper/MA45XDRX

@misc{pith2026250705403,
  author       = {Pith},
  title        = {Pith review of: PBE Meets LLM: When Few Examples Aren't Few-Shot Enough},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MA45XDRX}},
  note         = {Machine review of arXiv:2507.05403}
}
read the original abstract

Large language models (LLMs) can generate code from natural language descriptions. Their performance is typically evaluated using programming benchmarks that simulate real-world tasks. These benchmarks provide specifications in the form of docstrings, function signatures, or bug reports. The model then generates a program, which is tested against predefined test cases. In contrast, Programming by Example (PBE) uses input-output examples as the specification. Traditional PBE systems rely on search-based methods over restricted transformation spaces. They are usually designed for narrow domains and fixed input formats. It remains unclear how well LLMs perform on PBE tasks. In this work, we evaluate LLMs on PBE tasks involving tabular data transformations. We prompt models to generate functions that convert an input table to an output table. We test the generated functions on unseen inputs to measure accuracy. Our study includes multiple LLMs and evaluates different prompting strategies, such as one-shot vs. multi-try. We also compare performance with and without PBE-specific knowledge. Finally, we propose a hybrid method that calls a traditional PBE solver first, and then falls back to LLMs if necessary. Our results show that LLMs support more diverse input formats and achieve higher accuracy than conventional methods. However, they struggle with tasks that contain ambiguity. The hybrid approach improves overall success by combining the strengths of both approaches.

Figures

Figures reproduced from arXiv: 2507.05403 by the authors.

Figure 1
Figure 1. Overview of PBE Architecture[9] ranking accuracy, and correctly identifies intended programs 50% more effectively than heuristic-based methods [10]. This technique makes PBE more scalable, adaptive, and practical for real-world applications like data wrangling and code transformation. 2.1.3 Foofah. Taking inspiration from the classic A * algorithm [13], Foofha proposed a heuristic search algorithm to synthesize data… view at source ↗
Figure 2
Figure 2. Prompt Engineering Pipeline [35] Those methods enable models to be more closely tailored to spe￾cific tasks, improve accuracy, and reduce computational costs. This section focuses on prompt engineering, covering the key ideas and different approaches employed in this work. 2.2.1 Zero-shot prompting: refers to the model’s ability to solve the user query without needing examples during inference, eliminating the need … view at source ↗
Figure 3
Figure 3. Trends in code generation failures across iterative [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: GPT-4o extracts dates accurately but lacks common [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: The model incorrectly generalizes from the example [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 15 canonical work pages

  1. [1]

    Ilyas, Mourad Ouzzani, Paolo Papotti, and Michael Stonebraker

    Ziawasch Abedjan, John Morcos, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, and Michael Stonebraker. 2016. DataXFormer: A robust transformation discovery system. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE) . 1134–1145. https://doi.org/10.1109/ICDE.2016.7498319

    work page arXiv 2016

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. 2016. Ringer: web automation by demonstration. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (Amsterdam, Netherlands) (OOPSLA 2016) . Association for Computing Machinery, New York, NY, USA, 748–764. https:...

    arXiv 2016

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  5. [5]

    Guo, Robert DeLine, and Sumit Gulwani

    Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, and Sumit Gulwani. 2020. Wrex: A Unified Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. h...

    arXiv 2020

  6. [6]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually- crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861 (2023)

    arXiv 2023

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and etc. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    arXiv 2024

  8. [8]

    Sumit Gulwani. 2011. Automating String Processing in Spreadsheets Using Input- Output Examples. In Proceedings of the 38th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) . 317–330. https://doi.org/10. 1145/1926385.1926423

    arXiv 2011

Show all 42 references
  1. [9]

    Sumit Gulwani. 2015. Programming by Examples (and its applications in Data Wrangling). In Lectures at Marktoberdorf Summer School, Aug 2015 (lectures at marktoberdorf summer school, aug 2015 ed.). https://www.microsoft.com/en-us/ research/publication/programming-examples-appli...

    2015

  2. [10]

    Sumit Gulwani and Prateek Jain. 2017. Programming by Examples: PL meets ML. In APLAS 2017 (aplas 2017 ed.). Springer. https://www.microsoft.com/en- us/research/publication/programming-examples-pl-meets-ml/

    2017

  3. [11]

    Guo, Sean Kandel, Joseph M

    Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011. Proactive wrangling: mixed-initiative end-user programming of data transformation scripts. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, Calif...

    2011

  4. [12]

    Harris and Sumit Gulwani

    William R. Harris and Sumit Gulwani. 2011. Spreadsheet table transforma- tions from examples. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (San Jose, California, USA) (PLDI ’11). Association for Computing Machinery, New Yo...

    2011

  5. [13]

    Hart, Nils J

    Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A Formal Basis for the Heuristic Determination of Minimum Cost Paths. IEEE Transactions on Systems Science and Cybernetics 4, 2 (1968), 100–107. https://doi.org/10.1109/TSSC.1968. 300136

    1968 doi

  6. [14]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021)

    2021

  7. [15]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  8. [16]

    Anderson, Michael Cafarella, and H

    Zhongjun Jin, Michael R. Anderson, Michael Cafarella, and H. V. Jagadish. 2017. Foofah: Transforming Data By Example. In Proceedings of the 2017 ACM In- ternational Conference on Management of Data (Chicago, Illinois, USA) (SIG- MOD ’17). Association for Computing Machinery, N...

    2017

  9. [17]

    Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wran- gler: interactive visual specification of data transformation scripts. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (, Vancouver, BC, Canada,) (CHI ’11). Association...

    2011

  10. [18]

    Vu Le and Sumit Gulwani. 2014. FlashExtract: a framework for data extraction by examples. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (Edinburgh, United Kingdom) (PLDI ’14). Association for Computing Machinery, New York, ...

    2014

  11. [19]

    Douglas B. Lenat. 1976. AM: An Artificial Intelligence Approach to Discovery in Mathematics as Heuristic Search . Technical Report AIM-286. Stanford University, Artificial Intelligence Laboratory

    1976

  12. [20]

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024. Chain-of-Knowledge: Grounding Large Lan- guage Models via Dynamic Knowledge Adapting over Heterogeneous Sources. arXiv:2305.13269 [cs.CL] https://arxiv.org/abs/2305.13269

    2024 arXiv

  13. [21]

    Henry Lieberman (Ed.). 2001. Your Wish is My Command: Programming by Example. Morgan Kaufmann, San Francisco, CA

    2001

  14. [22]

    McKinley

    Na Meng, Miryung Kim, and Kathryn S. McKinley. 2011. Systematic editing: generating program transformations from an example. SIGPLAN Not. 46, 6 (June 2011), 329–342. https://doi.org/10.1145/1993316.1993537

    2011

  15. [23]

    Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai

  16. [24]

    Noor Nashid, Islem Bouzenia, Michael Pradel, and Ali Mesbah. 2025. Is- sue2Test: Generating Reproducing Test Cases from Issue Reports. arXiv preprint arXiv:2503.16320 (2025)

    2025

  17. [25]

    OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774

    2024 arXiv

  18. [26]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI (2019). https://cdn.openai.com/better-language-models/language_models_are_ unsupervised_multitask_learners.pdf Accessed: 2024-11-15

    2019

  19. [27]

    Vijayshankar Raman and Joseph M Hellerstein. 2001. Potter’s wheel: An interac- tive data cleaning system. In VLDB, Vol. 1. 381–390

    2001

  20. [28]

    Hellerstein

    Vijayshankar Raman and Joseph M. Hellerstein. 2001. Potter’s Wheel: An Inter- active Data Cleaning System. In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB ’01) . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 381–390

    2001

  21. [29]

    Mohammad Raza and Sumit Gulwani. 2017. Automated Data Extraction Using Predictive Program Synthesis. Proceedings of the AAAI Conference on Artificial Intelligence 31, 1 (Feb. 2017). https://doi.org/10.1609/aaai.v31i1.10668

    2017 doi

  22. [30]

    Ilyas, and Christopher Ré

    Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré

  23. [31]

    Jaeho Shin, Sen Wu, Feiran Wang, Christopher De Sa, Ce Zhang, and Christopher Ré. 2015. Incremental knowledge base construction using DeepDive. Proc. VLDB Endow. 8, 11 (July 2015), 1310–1321. https://doi.org/10.14778/2809974.2809991

    2015

  24. [32]

    Rishabh Singh. 2016. BlinkFill: semi-supervised programming by example for syntactic string transformations. Proc. VLDB Endow. 9, 10 (jun 2016), 816–827. https://doi.org/10.14778/2977797.2977807

    2016

  25. [33]

    Rishabh Singh and Sumit Gulwani. 2012. Learning Semantic String Transforma- tions from Examples. Proceedings of the VLDB Endowment (PVLDB) 5, 8 (2012), 740–751. https://doi.org/10.14778/2212351.2212357

    2012

  26. [34]

    Rishabh Singh and Sumit Gulwani. 2012. Synthesizing Number Transformations from Input-Output Examples. In Proceedings of the 24th International Conference on Computer Aided Verification (CA V). 634–651. https://doi.org/10.1007/978-3- 642-31424-7_44

    2012 doi

  27. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    2023 arXiv

  28. [36]

    Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017. Synthesizing Highly Expressive SQL Queries from Input-Output Examples. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 452–466. https://doi.org/10.1145/3062341.3062365

    2017

  29. [37]

    Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. 2025. Reinforcing Multi-Turn Reasoning in LLM Agents via Turn- Level Credit Assignment. arXiv:2505.11821 [cs.LG] https://arxiv.org/abs/2505. 11821

    2025

  30. [38]

    Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Ne- grevergne, and Gabriel Synnaeve. 2025. What Makes Large Language Models Reason in (Multi-Turn) Code Generation? arXiv:2410.08105 [cs.CL] https: //arxiv.org/abs/2410.08105

    2025 arXiv

  31. [39]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al

  32. [2013]

    In Pro- ceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research) , Sanjoy Dasgupta and David McAllester (Eds.), Vol

    A Machine Learning Framework for Programming by Example. In Pro- ceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research) , Sanjoy Dasgupta and David McAllester (Eds.), Vol. 28. PMLR, Atlanta, Georgia, USA, 187–195. https://p...

  33. [2017]

    arXiv:1702.00820 [cs.DB] https://arxiv.org/abs/1702.00820

    HoloClean: Holistic Data Repairs with Probabilistic Inference. arXiv:1702.00820 [cs.DB] https://arxiv.org/abs/1702.00820

    arXiv

  34. [2024]

    arXiv preprint arXiv:2406.15877 (2024)

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 (2024). 7

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.