{"id":"da1e474e-ffda-4bc0-a827-77afc92eaa3d","arxiv_id":"2412.14515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Vieira extends the Scallop relational engine with a foreign interface that lets foundation models act as probabilistic relations, enabling neuro-symbolic programs across nine tasks.","lead":"The paper presents Vieira, a declarative framework that treats foundation models like GPT, CLIP, and SAM as relations, so they can be combined with logic programs and probabilistic reasoning. It reports concise programs and competitive accuracy across nine language, vision, and retrieval tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation prompts are manually composed with knowledge of the test questions, so the reported accuracies may not generalize and the framework's advantage over direct prompting is confounded.","rationale":"The reader's weakest_assumption correctly identifies extraction robustness as the fragile point. I go one step further: the extraction prompts are not merely task-specific; they are explicitly composed to be similar to the test questions (stated in the DR task setup). This is a form of test-set contamination that makes the reported accuracies optimistic. If the prompts were instead selected from a held-out training split, the accuracy could drop, and the framework's generality would be in question. The concern is load-bearing because the paper's main evidence for \"viable general interface\" is the empirical performance on 9 tasks; if those numbers are inflated by test-aware prompt design, the central claim is unsupported. That said, the conceptual contribution—a foreign interface for treating foundation models as relational predicates—is credible and not invalidated by this concern. The framework itself may be useful; what is not established is the accuracy claim and the implied generality. Therefore the verdict should remain CONDITIONAL, with the condition that the evaluation be redone with proper train/test separation for prompt selection and with baselines that control for number of shots. This is in partial agreement with the reader: we identify the same fragile assumption but sharpen it to test-set-informed prompt design.","tokens_in":23384,"tokens_out":6246,"duration_ms":56011,"concrete_test":"Split each benchmark into training and test splits. For each relation-extraction predicate, compose the few-shot examples using only the training split (e.g., select examples by similarity to training questions, with no access to test questions). Run the full VIEIRA pipeline on the test split and compare accuracy to the reported numbers. If accuracy drops substantially (e.g., more than 5 points) on DR/TSO/CLUTRR/GSM8K, the reported gains reflect test-set-informed prompt engineering rather than the relational interface. Also run a direct-prompt baseline that uses the same few-shot examples in a single GPT-4 call (without symbolic reasoning) to isolate the contribution of the relational layer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Most load-bearing concern: the reported accuracy numbers are produced with few-shot extraction prompts that are manually composed with knowledge of the test dataset. In the Task setup for Date reasoning, the authors state: \"The shots for gpt_extract_relation are manually composed to be similar to questions in the dataset.\" This means the prompt examples were selected after seeing the test questions, so the extraction step is effectively tuned to the evaluation distribution. The same manual, dataset-specific prompt design is used for TSO, CLUTRR, GSM8K, and other tasks. As a result, the experiments do not measure how VIEIRA would perform on new inputs; they measure how well GPT-4 can extract relations when given examples drawn from the test distribution. The central claim that relational programming is a viable general interface for composing foundation models is not supported by this evidence. A related confound is that the baseline GPT-4 numbers are mostly 0-shot or generic CoT, while VIEIRA uses several manually selected shots; the observed accuracy gap could be due to the few-shot examples rather than to the relational layer. Both issues make the \"comparable or better accuracy\" claim broader than the data justify.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VIEIRA, a declarative relational programming framework that treats foundation models as stateless functions with relational inputs and outputs. The framework is implemented by extending the Scallop compiler with a foreign interface supporting foreign predicates and attributes, and it ships plugins for 12 foundation models including GPT, CLIP, SAM, OWL-ViT, and Stable Diffusion. The authors evaluate VIEIRA on nine tasks spanning date reasoning, tracking shuffled objects, kinship reasoning, math reasoning, question answering with retrieval, product search, compositional VQA, visual object tagging, and image generation/editing. They report that VIEIRA programs are concise and achieve comparable or better no-training accuracy than direct prompting and task-specific baselines.","tokens_in":23569,"tokens_out":5163,"duration_ms":43036,"significance":"If the framework performs as claimed, it would be a useful contribution: a single abstraction for composing foundation models with logic, probabilistic reasoning, and heterogeneous data, backed by concrete artifacts (the open-source compiler extension and plugin library). The paper's strengths include the breadth of the prototype, the plausibility of the relational abstraction, a fully implemented system, and honest disclosure of many evaluation details, including the fact that some prompts are manually adapted to the datasets. The main weakness is that the empirical evidence for the accuracy claim is not as carefully controlled as the system contribution: several prompts were written with knowledge of the test questions, baselines are not always comparable, and some evaluations rely on self-judgment or a model-as-judge. These issues do not invalidate the framework concept, but they do limit the strength of the comparative claims stated in the abstract and in RQ2.","major_comments":[{"comment":"The authors state that 'The shots for gpt_extract_relation are manually composed to be similar to questions in the dataset.' This means the extraction component was designed with knowledge of the test distribution for the DR task, and similar manual, dataset-specific prompt construction is reported for TSO, CLUTRR, GSM8K, and IGP20. Since the central RQ2 claim is that VIEIRA achieves comparable or better accuracy than competitive baselines, this evaluation does not measure generalization to new inputs; it measures performance when the few-shot examples are drawn from or tuned to the test distribution. This confound also affects the comparison with the 0-shot or generic CoT GPT-4 baselines in Table 3, because the observed gap could be due to the few-shot examples rather than to the relational layer. To support the claim, the authors should either develop prompts on a held-out development split and report test-set numbers, or re-frame the accuracy results as in-distribution prompt-engineering demonstrations rather than evidence of general framework superiority.","section":"Experimental Details, Task setup (Date reasoning)"},{"comment":"Table 3 reports no variance or significance tests, and the baseline configuration is inconsistent: DR and TSO use 0-shot or 0-shot CoT for GPT-4, CLUTRR uses 3-shot, and GSM8K uses 0-shot or 5-shot CoT. The comparison therefore does not hold the number of in-context examples fixed between VIEIRA and the baselines. Moreover, the claim of 'better performance than comparable baselines on 6 out of 8 studied datasets with baselines' is undercut by GSM8K, where VIEIRA (90.60) is below GPT-4 CoT (92.00), and by HotpotQA, where VIEIRA (67.3%) is below the fine-tuned baselines; it is not clear which six datasets remain after accounting for comparability. The authors should include confidence intervals, run baselines with matched few-shot examples, and state precisely which comparisons support the claim.","section":"Experiments and Analysis (RQ2), Table 3"},{"comment":"For HotpotQA, the paper says 'we use GPT-4 to check the correctness of the derived result against the ground truth.' Since the VIEIRA pipeline itself uses GPT-4 for QA, this is an automatic evaluation where the judge is the same model family as the answerer. The paper does not validate the LLM judge against human judgments on a sample, so the reported 67.3% EM could be biased. Please report human agreement on a subset or use an exact-match evaluator with normalized answers.","section":"Experimental Details, Retrieval augmentation (HotpotQA)"},{"comment":"The quantitative results in Table 6 for VOT and IGE rely on manual inspection by the authors on self-curated or small datasets (VQAR 100 items, OFCP 50 items, IGP20 20 items), without reporting inter-annotator agreement or a detailed rubric. Semantic correctness rates of 67.61%, 60.82%, and 74.00% are therefore not reproducible in a statistically meaningful way. Since these numbers are part of the paper's comparative claims, the authors should provide the number of evaluators, the annotation instructions, and ideally independent annotations.","section":"Experimental Details, Visual object tagging and Image generation and editing"}],"minor_comments":[{"comment":"The sentence 'From here, we just invoke the We continue to discuss how do we obtain the size and spatial relationships.' appears to be a copy-editing artifact; it should be rewritten or removed.","section":"Appendix, Detailed Example (CLEVR)"},{"comment":"The phrase 'On GQA, out method out-' is missing its completion; it should read 'our method outperforms'.","section":"Experiments and Analysis (RQ2), VQA paragraph"},{"comment":"The model setup mentions 'gpt-3-turbo', while the main text refers to 'gpt-3.5-turbo'; please clarify which model was actually used.","section":"Experimental Details, Model setup (GPT)"},{"comment":"The sentence 'We randomly sample 184 and 480' is incomplete; please specify that 184 and 480 refer to GQA and CLEVR samples, respectively.","section":"Experimental Details, Task setup (Compositional multi-modal reasoning)"},{"comment":"The sentence 'This py_eval function is wrapped as a foreign attribute in .' is missing the attribute name; please complete it.","section":"Appendix, Detailed Example (Math reasoning)"},{"comment":"The abstract and introduction list DALL-E as a model plugin, but the experiments do not use DALL-E; please clarify whether it is implemented but not evaluated or remove it from the examples.","section":"Introduction / Related Work"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is borderline for a top venue: the language/interface contribution is solid and the system is released, but the evaluation section overclaims what is demonstrated. The authors' disclosure that few-shot prompts were manually composed to be similar to test questions is commendable transparency, but that fact directly weakens the headline 'comparable or better accuracy' claim. The major revisions requested—held-out prompt development, matched baselines, validated LLM-judge, and richer manual-evaluation reporting—are feasible within the manuscript's scope. I do not see a fatal correctness flaw in the system design itself, so reject would be too harsh; at the same time, accept is not defensible given the gap between evidence and claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nQuick take: this is a credible systems paper that extends Scallop with a foreign interface for foundation models, but the accuracy comparisons are softer than the headline suggests—the few-shot extraction prompts were hand-composed for the test tasks and the baselines aren't shot-matched.\n\nWhat's genuinely new: the foreign predicate/attribute abstraction is clean, the plugin library (12 models) is real and open source, and the nine tasks show real breadth. The paper also demonstrates a nice property: the relational layer gives you interpretability and debugging for free. On qualitative examples like TSO (100%) and CLUTRR, the framework looks genuinely useful.\n\nThe soft spots are in the evaluation. The DR setup explicitly says the gpt_extract_relation shots were \"manually composed to be similar to questions in the dataset.\" That's a test-distribution leak: the extraction step is tuned to the test set, so the reported accuracies may not transfer to new inputs. The same hand-crafting likely applies to other tasks, though the paper is open about it only there. Compounding this, the GPT-4 baselines are mostly 0-shot or generic CoT, while Vieira uses 1–5 hand-picked shots. The accuracy gap could be due to the few-shot examples, not the relational layer. For a fair claim, you'd want shot-matched baselines.\n\nOther issues are more moderate: HotpotQA answers are judged by GPT-4—the same family of model in the pipeline—which is a mild circularity; two tasks (OFCP, IGP20) use manual inspection on self-curated data; and there are no error bars or significance tests anywhere. The DR benchmark was modified (28 of 369 questions corrected), which is fine if disclosed, and it is, but it makes the \"comparable or better\" claim narrower.\n\nNone of this kills the paper. The framework is a genuine contribution to neuro-symbolic programming, and the qualitative demonstrations are strong. But the quantitative claims need a revision pass before this should appear as-is.\n\nWho should read it: people working on neuro-symbolic systems, foundation-model orchestration, and logic-based DSLs. It deserves peer review—the implementation and interface design are real—but a serious referee should push for shot-matched baselines and a clearer separation of the relational contribution from prompt tuning.\n\nMy recommendation: send to a good venue, but with the expectation of substantial evaluation revisions.","headline":"Credible framework paper for relational programming with foundation models; accuracy claims overreach because few-shot prompts are hand-tuned to the test tasks and baselines aren't shot-matched.","tokens_in":24166,"tokens_out":3212,"would_cite":true,"duration_ms":27724,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating foundation models as stateless relations lets one declarative language combine GPT, CLIP, and SAM with logic and probability for diverse AI tasks.","keywords":["relational programming","foundation models","neuro-symbolic AI","probabilistic logic programming","Datalog","foreign interface","multi-modal composition","in-context learning"],"falsifier":"A concrete falsifier would be a task drawn from the same distribution where the few-shot extraction prompt fails on a nontrivial fraction of inputs (for example, kinship relations stated implicitly or with coreference) and the overall accuracy drops below direct GPT-4 prompting, showing the relational wrapper adds no robustness; alternatively, re-running the nine benchmarks with a weaker extraction model and demonstrating that accuracy collapses to near zero would show the framework's success is an artifact of GPT-4's extraction ability rather than the relational architecture.","tokens_in":23116,"feed_emoji":"🧩","tokens_out":5393,"duration_ms":41138,"temperature":0.7,"pith_summary":"The paper proposes Vieira, a declarative framework that treats foundation models as stateless functions with relational inputs and outputs. It argues that a probabilistic relational paradigm, the same style used in Datalog, can unify mechanisms like in-context learning, retrieval, and tool use into a single programming interface. Programs written in Vieira combine logic rules with calls to models such as GPT, CLIP, and SAM, and the framework reports comparable or better accuracy than direct prompting baselines on nine benchmarks spanning language, vision, and databases. The paper's central claim is that relational programming is a viable general interface for composing foundation models with deterministic logic, probabilistic reasoning, and heterogeneous data modalities.","feed_headline":"One language turns GPT, CLIP, and SAM into queryable relations","feed_subtitle":"Vieira combines logic, probability, and multi-modal models, matching or beating direct prompting on nine tasks.","key_machinery":"The load-bearing mechanism is the foreign interface consisting of foreign predicates (FPs) and foreign attributes (FAs). An FP is a function that takes bound arguments and returns a list of free-argument tuples, optionally with probabilities; an FA is a higher-order decorator that returns a new predicate, providing a clean way to specify model configuration and examples. This interface, combined with Scallop's provenance-based semiring for top-k proofs and the soft-eq operator on tensors, lets probabilistic and logical reasoning operate directly on model outputs, enabling features like ranking and uncertainty estimation. Algebraic data types supply domain-specific languages that bridge structured and unstructured data, allowing the same program to parse a natural-language question into a DSL and then evaluate it recursively.","core_discovery":"The central discovery is that a foundation model, regardless of whether it is a language model, a vision model, or a multi-modal model, can be abstracted as a stateless function that maps relational inputs to relational outputs, and this abstraction fits naturally into a probabilistic Datalog dialect. A foreign predicate calls a model and returns tuples tagged with probabilities; a foreign attribute configures the underlying model and decorates a predicate declaration, hiding low-level details. This lets a single program perform relational extraction with GPT, soft-join semantic search over embeddings, probabilistic classification with CLIP, recursive DSL evaluation for visual question answering, and composition of generative models for image editing, all under one declarative semantics with inspection of intermediate relations.","pith_inferences":["If the relational abstraction proves robust, it suggests a standard calling convention for foundation models analogous to SQL for databases, where applications are written once against a logical schema and the underlying model can be upgraded or replaced without rewriting the program.","The framework's reliance on hand-authored few-shot prompts for relation extraction is likely the bottleneck; a learned or more automated extraction step could extend the approach to open-domain inputs without per-task prompt engineering.","One testable extension would be to apply Vieira to tasks where the foundation model must be fine-tuned, since the current paper is limited to in-context learning; the relational structure could provide gradient paths through the logic.","The paper's image-editing DSL suggests that generative models can be composed as relations too, which may generalize to other generative pipelines where multi-step control is needed."],"forward_implications":["Vieira programs can replace task-specific glue code with a few dozen lines of declarative rules, making multi-model applications like retrieval-augmented QA and visual object tagging concise and inspectable.","Because the framework is no-training and model-agnostic, the same relational program can swap in different foundation models by changing a plugin, without altering the logic.","The probabilistic semantics carry model confidence into downstream reasoning, allowing uncertainty to propagate through logic programs rather than being discarded at the model output.","On the nine evaluated benchmarks, the relational solutions match or exceed direct GPT-4 prompting and several fine-tuned baselines, with gains on tasks requiring systematic multi-step reasoning like date arithmetic and object tracking."],"supporting_citations":[{"why":"Supplies the Scallop compiler and provenance-based semiring framework that Vieira extends with its foreign interface.","marker":"(Li, Huang, and Naik 2023)"},{"why":"Establishes the probabilistic relational programming paradigm that Vieira follows and builds upon.","marker":"(Manhaeve et al. 2018)"},{"why":"Provides the compositional visual reasoning approach and DSL that Vieira adapts for GQA, CLEVR, and image editing.","marker":"(Gupta and Kembhavi 2022)"},{"why":"Represents the program-aided language model mechanism that Vieira unifies with retrieval and in-context learning.","marker":"(Gao et al. 2023)"},{"why":"Offers CLIP, the image-text alignment model used through the foreign interface for zero-shot classification and semantic search.","marker":"(Radford et al. 2021)"},{"why":"Names GPT-4, the model used for relation extraction, semantic parsing, and question answering across most tasks.","marker":"(OpenAI 2023)"},{"why":"Supplies GSM8K, the math word problem benchmark used to compare Vieira's DSL-evaluation approach against direct prompting.","marker":"(Cobbe et al. 2021)"}],"fun_headline_variants":["Vieira treats GPT, CLIP, and SAM as queryable relations","One language to relate GPT, CLIP, and SAM","Vieira: Foundation models as relational building blocks","The relational way to program with foundation models","Logic-driven AI composition: Vieira's relational twist"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that few-shot prompting of a large language model will reliably convert free text into the structured relations the symbolic reasoner consumes; if that extraction step falters on new inputs, the downstream logic cannot recover the answer.","fun_headline_variants_meta":{"raw":{"variants":["Vieira treats GPT, CLIP, and SAM as queryable relations","One language to relate GPT, CLIP, and SAM","Vieira: Foundation models as relational building blocks","The relational way to program with foundation models","Logic-driven AI composition: Vieira's relational twist"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3041,"prompt_tokens":865,"completion_tokens":2176,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2096}},"tokens_in":481,"tokens_out":2176,"duration_ms":14057,"temperature":1.0,"reasoning_tokens":2096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:09:55.449033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier would be a task drawn from the same distribution where the few-shot extraction prompt fails on a nontrivial fraction of inputs (for example, kinship relations stated implicitly or with coreference) and the overall accuracy drops below direct GPT-4 prompting, showing the relational wrapper adds no robustness; alternatively, re-running the nine benchmarks with a weaker extraction model and demonstrating that accuracy collapses to near zero would show the framework's success is an artifact of GPT-4's extraction ability rather than the relational architecture.","supporting_citations":[],"review_version":1}