Pith. sign in

REVIEW 4 major objections 6 minor 57 references

Relational Programming with Foundation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating foundation models as stateless relations lets one declarative language combine GPT, CLIP, and SAM with logic and probability for diverse AI tasks.

desk verdict Credible framework paper for relational programming with foundation models; accuracy claims overreach because few-shot prompts are hand-tuned to the test tasks and baselines aren't shot-matched. read the letter →

arxiv 2412.14515 v1 pith:EH5GYQQN submitted 2024-12-19 cs.AI cs.PL

Ziyang Li , Jiani Huang , Jason Liu , Felix Zhu , Eric Zhao , William Dodds , Neelay Velingker , Rajeev Alur
show 1 more author
Mayur Naik
This is my paper · ORCID
classification cs.AIcs.PL
keywords relationalprogrammingfoundationmodelsneuro-symbolicAIprobabilisticlogicDatalogforeigninterfacemulti-modalcompositionin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Vieira, a declarative framework that treats foundation models as stateless functions with relational inputs and outputs. It argues that a probabilistic relational paradigm, the same style used in Datalog, can unify mechanisms like in-context learning, retrieval, and tool use into a single programming interface. Programs written in Vieira combine logic rules with calls to models such as GPT, CLIP, and SAM, and the framework reports comparable or better accuracy than direct prompting baselines on nine benchmarks spanning language, vision, and databases. The paper's central claim is that relational programming is a viable general interface for composing foundation models with deterministic logic, probabilistic reasoning, and heterogeneous data modalities.

What carries the argument

The load-bearing mechanism is the foreign interface consisting of foreign predicates (FPs) and foreign attributes (FAs). An FP is a function that takes bound arguments and returns a list of free-argument tuples, optionally with probabilities; an FA is a higher-order decorator that returns a new predicate, providing a clean way to specify model configuration and examples. This interface, combined with Scallop's provenance-based semiring for top-k proofs and the soft-eq operator on tensors, lets probabilistic and logical reasoning operate directly on model outputs, enabling features like ranking and uncertainty estimation. Algebraic data types supply domain-specific languages that bridge structured and unstructured data, allowing the same program to parse a natural-language question into a DSL and then evaluate it recursively.

What would settle it

A concrete falsifier would be a task drawn from the same distribution where the few-shot extraction prompt fails on a nontrivial fraction of inputs (for example, kinship relations stated implicitly or with coreference) and the overall accuracy drops below direct GPT-4 prompting, showing the relational wrapper adds no robustness; alternatively, re-running the nine benchmarks with a weaker extraction model and demonstrating that accuracy collapses to near zero would show the framework's success is an artifact of GPT-4's extraction ability rather than the relational architecture.

Watch

Extended reading notes

Core claim

The central discovery is that a foundation model, regardless of whether it is a language model, a vision model, or a multi-modal model, can be abstracted as a stateless function that maps relational inputs to relational outputs, and this abstraction fits naturally into a probabilistic Datalog dialect. A foreign predicate calls a model and returns tuples tagged with probabilities; a foreign attribute configures the underlying model and decorates a predicate declaration, hiding low-level details. This lets a single program perform relational extraction with GPT, soft-join semantic search over embeddings, probabilistic classification with CLIP, recursive DSL evaluation for visual question answering, and composition of generative models for image editing, all under one declarative semantics with inspection of intermediate relations.

Load-bearing premise

The load-bearing premise is that few-shot prompting of a large language model will reliably convert free text into the structured relations the symbolic reasoner consumes; if that extraction step falters on new inputs, the downstream logic cannot recover the answer.

Editorial extensions

If this is right

  • Vieira programs can replace task-specific glue code with a few dozen lines of declarative rules, making multi-model applications like retrieval-augmented QA and visual object tagging concise and inspectable.
  • Because the framework is no-training and model-agnostic, the same relational program can swap in different foundation models by changing a plugin, without altering the logic.
  • The probabilistic semantics carry model confidence into downstream reasoning, allowing uncertainty to propagate through logic programs rather than being discarded at the model output.
  • On the nine evaluated benchmarks, the relational solutions match or exceed direct GPT-4 prompting and several fine-tuned baselines, with gains on tasks requiring systematic multi-step reasoning like date arithmetic and object tracking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the relational abstraction proves robust, it suggests a standard calling convention for foundation models analogous to SQL for databases, where applications are written once against a logical schema and the underlying model can be upgraded or replaced without rewriting the program.
  • The framework's reliance on hand-authored few-shot prompts for relation extraction is likely the bottleneck; a learned or more automated extraction step could extend the approach to open-domain inputs without per-task prompt engineering.
  • One testable extension would be to apply Vieira to tasks where the foundation model must be fine-tuned, since the current paper is limited to in-context learning; the relational structure could provide gradient paths through the logic.
  • The paper's image-editing DSL suggests that generative models can be composed as relations too, which may generalize to other generative pipelines where multi-step control is needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VIEIRA, a declarative relational programming framework that treats foundation models as stateless functions with relational inputs and outputs. The framework is implemented by extending the Scallop compiler with a foreign interface supporting foreign predicates and attributes, and it ships plugins for 12 foundation models including GPT, CLIP, SAM, OWL-ViT, and Stable Diffusion. The authors evaluate VIEIRA on nine tasks spanning date reasoning, tracking shuffled objects, kinship reasoning, math reasoning, question answering with retrieval, product search, compositional VQA, visual object tagging, and image generation/editing. They report that VIEIRA programs are concise and achieve comparable or better no-training accuracy than direct prompting and task-specific baselines.

Significance. If the framework performs as claimed, it would be a useful contribution: a single abstraction for composing foundation models with logic, probabilistic reasoning, and heterogeneous data, backed by concrete artifacts (the open-source compiler extension and plugin library). The paper's strengths include the breadth of the prototype, the plausibility of the relational abstraction, a fully implemented system, and honest disclosure of many evaluation details, including the fact that some prompts are manually adapted to the datasets. The main weakness is that the empirical evidence for the accuracy claim is not as carefully controlled as the system contribution: several prompts were written with knowledge of the test questions, baselines are not always comparable, and some evaluations rely on self-judgment or a model-as-judge. These issues do not invalidate the framework concept, but they do limit the strength of the comparative claims stated in the abstract and in RQ2.

major comments (4)
  1. [Experimental Details, Task setup (Date reasoning)] The authors state that 'The shots for gpt_extract_relation are manually composed to be similar to questions in the dataset.' This means the extraction component was designed with knowledge of the test distribution for the DR task, and similar manual, dataset-specific prompt construction is reported for TSO, CLUTRR, GSM8K, and IGP20. Since the central RQ2 claim is that VIEIRA achieves comparable or better accuracy than competitive baselines, this evaluation does not measure generalization to new inputs; it measures performance when the few-shot examples are drawn from or tuned to the test distribution. This confound also affects the comparison with the 0-shot or generic CoT GPT-4 baselines in Table 3, because the observed gap could be due to the few-shot examples rather than to the relational layer. To support the claim, the authors should either develop prompts on a held-out development split and report test-set numbers, or re-frame the accuracy results as in-distribution prompt-engineering demonstrations rather than evidence of general framework superiority.
  2. [Experiments and Analysis (RQ2), Table 3] Table 3 reports no variance or significance tests, and the baseline configuration is inconsistent: DR and TSO use 0-shot or 0-shot CoT for GPT-4, CLUTRR uses 3-shot, and GSM8K uses 0-shot or 5-shot CoT. The comparison therefore does not hold the number of in-context examples fixed between VIEIRA and the baselines. Moreover, the claim of 'better performance than comparable baselines on 6 out of 8 studied datasets with baselines' is undercut by GSM8K, where VIEIRA (90.60) is below GPT-4 CoT (92.00), and by HotpotQA, where VIEIRA (67.3%) is below the fine-tuned baselines; it is not clear which six datasets remain after accounting for comparability. The authors should include confidence intervals, run baselines with matched few-shot examples, and state precisely which comparisons support the claim.
  3. [Experimental Details, Retrieval augmentation (HotpotQA)] For HotpotQA, the paper says 'we use GPT-4 to check the correctness of the derived result against the ground truth.' Since the VIEIRA pipeline itself uses GPT-4 for QA, this is an automatic evaluation where the judge is the same model family as the answerer. The paper does not validate the LLM judge against human judgments on a sample, so the reported 67.3% EM could be biased. Please report human agreement on a subset or use an exact-match evaluator with normalized answers.
  4. [Experimental Details, Visual object tagging and Image generation and editing] The quantitative results in Table 6 for VOT and IGE rely on manual inspection by the authors on self-curated or small datasets (VQAR 100 items, OFCP 50 items, IGP20 20 items), without reporting inter-annotator agreement or a detailed rubric. Semantic correctness rates of 67.61%, 60.82%, and 74.00% are therefore not reproducible in a statistically meaningful way. Since these numbers are part of the paper's comparative claims, the authors should provide the number of evaluators, the annotation instructions, and ideally independent annotations.
minor comments (6)
  1. [Appendix, Detailed Example (CLEVR)] The sentence 'From here, we just invoke the We continue to discuss how do we obtain the size and spatial relationships.' appears to be a copy-editing artifact; it should be rewritten or removed.
  2. [Experiments and Analysis (RQ2), VQA paragraph] The phrase 'On GQA, out method out-' is missing its completion; it should read 'our method outperforms'.
  3. [Experimental Details, Model setup (GPT)] The model setup mentions 'gpt-3-turbo', while the main text refers to 'gpt-3.5-turbo'; please clarify which model was actually used.
  4. [Experimental Details, Task setup (Compositional multi-modal reasoning)] The sentence 'We randomly sample 184 and 480' is incomplete; please specify that 184 and 480 refer to GQA and CLEVR samples, respectively.
  5. [Appendix, Detailed Example (Math reasoning)] The sentence 'This py_eval function is wrapped as a foreign attribute in .' is missing the attribute name; please complete it.
  6. [Introduction / Related Work] The abstract and introduction list DALL-E as a model plugin, but the experiments do not use DALL-E; please clarify whether it is implemented but not evaluated or remove it from the examples.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the framework is evaluated on external benchmarks, and the mild evaluation confounds do not reduce the central claim to its inputs.

full rationale

The paper does not derive its accuracy claims from its assumptions; it implements a compiler extension and measures performance on external benchmarks (BIG-bench, CLUTRR, GSM8K, HotpotQA, ESCI, GQA, CLEVR, VQAR, OFCP, IGP20). There are no fitted parameters renamed as predictions, no uniqueness theorem imported from self-citation, and no definitional equivalence between inputs and outputs. The Scallop citation is to an existing open-source system that VIEIRA extends, and it is not used to forbid alternatives or to supply an unverified load-bearing premise. The closest concerns are measurement confounds rather than circular derivation. First, HotpotQA answers are checked using GPT-4, the same model family used in the solution: the paper states 'we use GPT-4 to check the correctness of the derived result against the ground truth.' Second, the DR ground truth was corrected by the authors: '28 of the original 369 questions being corrected for wrong target answers.' Third, few-shot prompts are manually composed with knowledge of the test distribution: 'The shots for gpt_extract_relation are manually composed to be similar to questions in the dataset.' These issues affect how strongly the empirical results generalize and how fairly they compare to baselines, but they do not make the central claim equivalent to its inputs. No equation or fitted parameter reduces to the reported results, so no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces programming abstractions (foreign predicates and foreign attributes) but no new physical or conceptual entities such as particles, forces, or dimensions. These abstractions are implemented as code artifacts, as shown in Figure 2 and the Foreign Interface section, so the invented-entity ledger is empty.

free parameters (5)
  • CLIP face-tagging probability threshold = 0.8
    Hand-set threshold in the OFCP face tagging pipeline; faces with CLIP probability below 0.8 are tagged unknown, directly affecting measured accuracy.
  • ViLT top-k and score threshold = top=5, score_threshold=0.1
    Hand-set in the VQAR and OFCP pipelines; controls which candidate answers from ViLT are retained for relational reasoning.
  • DSFD confidence and NMS thresholds = confidence_threshold=0.5, nms_iou_threshold=0.3
    Chosen in the face detection setup; affects which faces are detected and passed to CLIP for tagging.
  • OWL-ViT score threshold and crop parameters = score_threshold=0.1, expand_crop_region=0 or 10, limit=10
    Hand-set in VQA and object tagging; determines which segments are produced and whether raw confidence scores are flattened to 1.
  • Few-shot example counts per task = 1, 2, or 5 depending on task
    Prompts and example counts are manually composed per dataset; extraction quality is sensitive to these choices, and no sensitivity analysis is reported.
assumptions (4)
  • standard math Datalog semantics with the top-k-proofs semiring as implemented in Scallop is sound for the probabilistic reasoning used in this paper.
    The framework relies on Scallop's provenance framework (Huang et al. 2021; Li et al. 2023) for probability computation; the paper does not re-derive this.
  • domain assumption Pretrained foundation models produce sufficiently calibrated probability estimates over relational outputs to be treated as tuple probabilities.
    Used throughout, for example @clip returns probabilities and OWL-ViT flatten_probability=true overwrites confidence to 1 in CLEVR, showing the assumption is violated and hand-patched.
  • ad hoc to paper Author-written few-shot prompts and DSLs are representative of a general programming interface and transfer across tasks.
    Each task defines a bespoke DSL and hand-composed examples, such as the Query DSL for CLEVR and the kinship knowledge base for CLUTRR; no evidence of generalization beyond these prompts is provided.
  • domain assumption GPT-4 API outputs are deterministic enough at temperature 0 for evaluation to be stable.
    Experiments were run from June to August 2023 with temperature 0 and manual re-query on server errors, but no repeated-run variance is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relational Programming with Foundation Models." pith.science (2026). https://pith.science/paper/EH5GYQQN

@misc{pith2026241214515,
  author       = {Pith},
  title        = {Pith review of: Relational Programming with Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EH5GYQQN}},
  note         = {Machine review of arXiv:2412.14515}
}
read the original abstract

Foundation models have vast potential to enable diverse AI applications. The powerful yet incomplete nature of these models has spurred a wide range of mechanisms to augment them with capabilities such as in-context learning, information retrieval, and code interpreting. We propose Vieira, a declarative framework that unifies these mechanisms in a general solution for programming with foundation models. Vieira follows a probabilistic relational paradigm and treats foundation models as stateless functions with relational inputs and outputs. It supports neuro-symbolic applications by enabling the seamless combination of such models with logic programs, as well as complex, multi-modal applications by streamlining the composition of diverse sub-models. We implement Vieira by extending the Scallop compiler with a foreign interface that supports foundation models as plugins. We implement plugins for 12 foundation models including GPT, CLIP, and SAM. We evaluate Vieira on 9 challenging tasks that span language, vision, and structured and vector databases. Our evaluation shows that programs in Vieira are concise, can incorporate modern foundation models, and have comparable or better accuracy than competitive baselines.

Figures

Figures reproduced from arXiv: 2412.14515 by the authors.

Figure 1
Figure 1. Programs in VIEIRA using foundation models. Various mechanisms have been proposed to augment foundation models to overcome these limitations. For exam￾ple, PAL (Gao et al. 2023), WebGPT (Nakano et al. 2021), and Toolformer (Schick et al. 2023) connect LMs with search engines and external tools, expanding their informa￾tion retrieval and structural reasoning capabilities. LMQL (Beurer-Kellner, Fischer, and Vechev 202… view at source ↗
Figure 2
Figure 2. Snippet of Python implementation of the foreign [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Table 1 summarizes the datasets, evaluation metrics, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 3
Figure 3. Figure 3: Benchmark tasks. The top of each box lists the dataset(s) and the foundation models used in our solutions. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: Illustrative comparisons between our solution and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Systematic generalizability comparisons on the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Surface Syntax of VIEIRA language. NL question to programmatic query We use the GPT-4 model (OpenAI 2023) for converting a natural language question into a programmatic query. The first step is defining the domain specific language (DSL) for querying the CLEVR dataset:…
Figure 8
Figure 8. Figure 8: A CLEVR example data-point. parsed_query(q) and eval_bool(q, r) A concrete example We illustrate a concrete example in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Face Tagging (OFCP) exemplars [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Object Tagging (VQAR) exemplars [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Image Editing (OFCP) exemplars [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Image Generation and Editing (IGP20) exemplars. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 21 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abiteboul, S.; Hull, R.; and Vianu, V. 1994. Foundations of Databases: T he Logical Level . Pearson, 1st edition

    work page 1994

  4. [4]

    Adadi, A. 2021. A survey on data-efficient algorithms in big data era. Journal of Big Data, 8(1): 24

    work page 2021

  5. [5]

    Beurer-Kellner, L.; Fischer, M.; and Vechev, M. 2022. Prompting Is Programming: A Query Language For Large Language Models. In PLDI

    work page 2022

  6. [6]

    A.; Adeli, E.; Altman, R

    Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R. B.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bosselut, A.; Brunskill, E.; and et al. 2021. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258

    arXiv 2021

  7. [7]

    Brooks, T.; Holynski, A.; and Efros, A. A. 2023. InstructPix2Pix: Learning to Follow Image Editing Instructions. arXiv:2211.09800

    arXiv 2023

  8. [8]

    T.; Li, Y.; Lundberg, S.; et al

    Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv:2303.12712

    arXiv 2023

Show all 57 references
  1. [9]

    W.; Zhou, D.; Song, D.; and Le, Q

    Chen, X.; Liang, C.; Yu, A. W.; Zhou, D.; Song, D.; and Le, Q. V. 2020. Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension. In ICLR

    2020

  2. [10]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168

    2021 arXiv

  3. [11]

    Davis, E.; and Aaronson, S. 2023. Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems. arXiv:2308.05713

    2023 arXiv

  4. [12]

    Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. PAL: Program-aided Language Models. arXiv:2211.10435

    2023 arXiv

  5. [13]

    Gupta, T.; and Kembhavi, A. 2022. Visual Programming: Compositional visual reasoning without training. arXiv:2211.11559

    2022 arXiv

  6. [14]

    Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022. Prompt-to-Prompt Image Editing with Cross Attention Control. arXiv:2208.01626

    2022 arXiv

  7. [15]

    Huang, J.; Li, Z.; Chen, B.; Samel, K.; Naik, M.; Song, L.; and Si, X. 2021. Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning. In NeurIPS

    2021

  8. [16]

    A.; and Manning, C

    Hudson, D. A.; and Manning, C. D. 2019. GQA: a new dataset for compositional question answering over real-world images. arXiv:1902.09506

    2019 arXiv

  9. [17]

    L.; and Girshick, R

    Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei - Fei, L.; Zitnick, C. L.; and Girshick, R. B. 2016. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. arXiv:1612.06890

    2016 arXiv

  10. [18]

    Kim, W.; Son, B.; and Kim, I. 2021. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. arXiv:2102.03334

    2021 arXiv

  11. [19]

    C.; Lo, W.-Y.; et al

    Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023. Segment Anything. arXiv:2304.02643

    2023 arXiv

  12. [20]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. In NeurIPS

    2022

  13. [21]

    Li, J.; Wang, Y.; Wang, C.; Tai, Y.; Qian, J.; Yang, J.; Wang, C.; Li, J.; and Huang, F. 2018. DSFD: Dual Shot Face Detector. arXiv:1810.10220

    2018 arXiv

  14. [22]

    N.; and Zhu, S.-C

    Li, Q.; Huang, S.; Hong, Y.; Chen, Y.; Wu, Y. N.; and Zhu, S.-C. 2020. Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning. In ICML

    2020

  15. [23]

    Li, X.-Y.; Lei, W.-J.; and Yang, Y.-B. 2022. From Easy to Hard: Two-stage Selector and Reader for Multi-hop Question Answering. arXiv:2205.11729

    2022 arXiv

  16. [24]

    Li, Z.; Huang, J.; and Naik, M. 2023. Scallop: A Language for Neurosymbolic Programming. In PLDI

    2023

  17. [25]

    Liang, Y.; Wu, C.; Song, T.; Wu, W.; Xia, Y.; Liu, Y.; Ou, Y.; Lu, S.; Ji, L.; Mao, S.; Wang, Y.; Shou, L.; Gong, M.; and Duan, N. 2023. TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs. arXiv:2303.16434

    2023 arXiv

  18. [26]

    Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692

    2019 arXiv

  19. [27]

    Lyu, Q.; Havaldar, S.; Stein, A.; Zhang, L.; Rao, D.; Wong, E.; Apidianaki, M.; and Callison-Burch, C. 2023. Faithful Chain-of-Thought Reasoning. arXiv:2301.13379

    2023 arXiv

  20. [28]

    Manhaeve, R.; Dumancic, S.; Kimmig, A.; Demeester, T.; and Raedt, L. D. 2018. DeepProbLog: Neural Probabilistic Logic Programming. arXiv:1805.10872

    2018 arXiv

  21. [29]

    B.; and Wu, J

    Mao, J.; Gan, C.; Kohli, P.; Tenenbaum, J. B.; and Wu, J. 2019. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. arXiv:1904.12584

    2019 arXiv

  22. [30]

    J.; Johnson, M.; and Steedman, M

    McKenna, N.; Li, T.; Cheng, L.; Hosseini, M. J.; Johnson, M.; and Steedman, M. 2023. Sources of Hallucination by Large Language Models on Inference Tasks. arXiv:2305.14552

    2023 arXiv

  23. [31]

    Minderer, M.; Gritsenko, A.; Stone, A.; Neumann, M.; Weissenborn, D.; Dosovitskiy, A.; Mahendran, A.; Arnab, A.; Dehghani, M.; Shen, Z.; Wang, X.; Zhai, X.; Kipf, T.; and Houlsby, N. 2022. Simple Open-Vocabulary Object Detection with Vision Transformers. arXiv:2205.06230

    2022 arXiv

  24. [32]

    Minervini, P.; Riedel, S.; Stenetorp, P.; Grefenstette, E.; and Rockt \"a schel, T. 2020. Learning Reasoning Strategies in End-to-End Differentiable Proving. In ICML

    2020

  25. [33]

    Nakano, R.; Hilton, J.; Balaji, S.; Wu, J.; Ouyang, L.; Kim, C.; Hesse, C.; Jain, S.; Kosaraju, V.; Saunders, W.; Jiang, X.; Cobbe, K.; Eloundou, T.; Krueger, G.; Button, K.; Knight, M.; Chess, B.; and Schulman, J. 2021. WebGPT: Browser-assisted question-answering with human f...

    2021 arXiv

  26. [34]

    Nogueira, R.; and Cho, K. 2019. Passage Re-ranking with BERT. arXiv:1901.04085

    2019 arXiv

  27. [35]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774

    2023 arXiv

  28. [36]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  29. [37]

    Rajasekharan, A.; Zeng, Y.; Padalkar, P.; and Gupta, G. 2023. Reliable Natural Language Understanding with Large Language Models and Answer Set Programming. In International Conference on Logic Programming

    2023

  30. [38]

    Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021. Zero-shot text-to-image generation. In ICML

    2021

  31. [39]

    Ratner, N.; Levine, Y.; Belinkov, Y.; Ram, O.; Magar, I.; Abend, O.; Karpas, E.; Shashua, A.; Leyton-Brown, K.; and Shoham, Y. 2023. Parallel Context Windows for Large Language Models. In Proceedings of the ACL

    2023

  32. [40]

    K.; Màrquez, L.; Valero, F.; Rao, N.; Zaragoza, H.; Bandyopadhyay, S.; Biswas, A.; Xing, A.; and Subbian, K

    Reddy, C. K.; Màrquez, L.; Valero, F.; Rao, N.; Zaragoza, H.; Bandyopadhyay, S.; Biswas, A.; Xing, A.; and Subbian, K. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588

    2022 arXiv

  33. [41]

    Richards, T. B. 2023. AutoGPT . https://github.com/Significant-Gravitas/AutoGPT. Accessed: 2024-02-12

    2023

  34. [42]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In CVPR

    2022

  35. [43]

    Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761

    2023 arXiv

  36. [44]

    Sinha, K.; Sodhani, S.; Dong, J.; Pineau, J.; and Hamilton, W. L. 2019. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. arXiv:1908.06177

    2019 arXiv

  37. [45]

    Song, K.; Tan, X.; Qin, T.; Lu, J.; and Liu, T.-Y. 2020. MPNet: Masked and Permuted Pre-training for Language Understanding. arXiv:2004.09297

    2020 arXiv

  38. [46]

    Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; and et al. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615

    2023 arXiv

  39. [47]

    Tiong, A. M. H.; Li, J.; Li, B.; Savarese, S.; and Hoi, S. C. 2022. Plug-and-Play VQA : Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the ACL: EMNLP

    2022

  40. [48]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288

    2023 arXiv

  41. [49]

    L.; Wilder, B.; and Kolter, Z

    Wang, P.-W.; Donti, P. L.; Wilder, B.; and Kolter, Z. 2019. SATNet: Bridging Deep Learning and Logical Reasoning Using a Differentiable Satisfiability Solver. In ICML

    2019

  42. [50]

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171

    2023 arXiv

  43. [51]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903

    2023 arXiv

  44. [52]

    S.; Wong, Y.; Kankanhalli, M.; and Shah, M

    Xu, Z.; Rawat, Y. S.; Wong, Y.; Kankanhalli, M.; and Shah, M. 2022. Don't Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation. In NeurIPS

    2022

  45. [53]

    W.; Salakhutdinov, R.; and Manning, C

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. arXiv:1809.09600

    2018 arXiv

  46. [54]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629

    2023 arXiv

  47. [55]

    Yi, K.; Wu, J.; Gan, C.; Torralba, A.; Kohli, P.; and Tenenbaum, J. 2018. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS

    2018

  48. [56]

    Yin, Z.; Wang, Y.; Wu, Y.; Yan, H.; Hu, X.; Zhang, X.; Cao, Z.; Huang, X.; and Qiu, X. 2022. Rethinking Label Smoothing on Multi-hop Question Answering. arXiv:2212.09512

    2022 arXiv

  49. [57]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.