{"id":"8540e8d4-a10e-477f-9e3a-474a48698eb9","arxiv_id":"2509.07122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A facet-based comparison of DeepProbLog, Scallop, and DomiKnowS with efficiency measurements on four toy tasks, identifying challenges for future neurosymbolic frameworks.","lead":"This paper compares three neuro-symbolic AI frameworks (DeepProbLog, Scallop, and DomiKnowS) by breaking them into five technical components and running four small test tasks. It gives developers a map of what each framework is good at, but it reports speed and memory only, not accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 timing comparisons are taken on a single example without reporting accuracy, so the empirical tradeoff claims are not yet supported.","rationale":"Reading in good faith, the paper makes a useful conceptual contribution: the five-facet characterization in Sections 2 through 7 is concrete, grounded in code examples, and accompanied by a public repository, which supports reproducibility. The quantitative side, however, is where the central claim's second half lives, and it is under-evidenced. The reader's weakest assumption correctly identifies representativeness and implementation artifacts as the core risk. My analysis sharpens that concern by locating specific evidence in the manuscript itself: Section 6 restricts Table 2 to a single training/testing example, Table 2 omits accuracy entirely, many values lack error bars, and the one headline training-time difference on MNIST Sum is within the reported standard deviations. These are not disagreements with outside consensus; they are measurement-protocol problems that prevent the empirical claims from being established even on the authors' own terms. The fact that the authors are also developers of DomiKnowS makes the representativeness assumption more fragile, but no allegation of bias is needed: the protocol itself is insufficient. The proposed check - train to matched accuracy, measure over full epochs with multiple seeds, and report confidence intervals - would settle whether the claimed tradeoffs are framework-level or measurement artifacts. The reader's CONDITIONAL verdict remains appropriate; the concern reinforces it rather than changing it.","tokens_in":21078,"tokens_out":6277,"duration_ms":58726,"concrete_test":"Reproduce Table 2 with a fixed protocol: for each task, train all frameworks to a predetermined target accuracy (e.g., at least 95% on MNIST Sum and at least 90% on the other three) using the same neural backbone, dataset split, optimizer, and hardware; measure full-epoch training time and full-test inference time over at least 5 seeds, reporting accuracy, mean, and 95% confidence intervals. If DeepProbLog's training advantage over Scallop vanishes or flips once all runs are matched for accuracy and measured over complete epochs rather than a single example, the claimed empirical tradeoff is an artifact of the measurement protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical contribution (Section 6, Table 2) is billed as revealing measurable framework-level differences, including Scallop's fastest inference and DeepProbLog's slightly faster training. The load-bearing step is that these wall-clock and memory numbers reflect properties of the frameworks rather than of the particular implementations, tasks, and measurement protocol. The manuscript itself undercuts this: Section 6 says Table 2 compares efficiency 'on a single training/testing example.' With per-sample times in milliseconds, one-off overheads (library import, JIT/compilation, graph construction, solver licensing, garbage collection) can dominate and are not representative of end-to-end use. No accuracy is reported for any task, so a reader cannot tell whether each implementation actually learned the task; a fast-but-wrong or under-trained run would still enter Table 2. Error bars that are present overlap: MNIST Sum training time is 5.84 +/- 3.24 ms for DeepProbLog versus 6.50 +/- 2.35 ms for Scallop, making the 'slightly faster training' claim not significant even on the paper's own numbers. Because the empirical section is the only support for the claim that the four tasks reveal measurable differences, this concern is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conceptual characterization of neuro-symbolic (NeSy) frameworks along five facets: symbolic knowledge representation, neural modeling flexibility, model declaration, the interplay between symbolic and sub-symbolic components, and the use of large language models. It illustrates the taxonomy with three frameworks (DeepProbLog, Scallop, DomiKnowS) and adds LEFT for one visual question answering task. The authors provide code-level examples for four tasks (MNIST Sum, Shapes, Toy NER, and Math Equation Inference) and report wall-clock time and memory measurements in Table 2. The stated contributions are the identification of the five facets, a cross-framework comparison, and a list of requirements and challenges for future NeSy frameworks.","tokens_in":21238,"tokens_out":7224,"duration_ms":60873,"significance":"If the characterization is accepted, it offers a useful organizing vocabulary for a fragmented field and a reasonable starting point for principled framework comparison. The paper is strongest in its qualitative sections: the five-facet schema is coherent, the code snippets in Figures 2 and 3 make framework differences concrete, and the public repository of examples is a practical resource. The empirical part is the weakest element: the current measurements do not support the claimed framework-level efficiency tradeoffs because accuracy is not reported, statistical power is very low, and implementation differences are uncontrolled. A redesigned empirical study, or a clear framing of the paper as a purely conceptual contribution with anecdotal illustrations, would make the contribution more defensible.","major_comments":[{"comment":"The empirical comparison does not support the paper's central claim that the four tasks reveal measurable differences between frameworks. The text states that Table 2 compares efficiency 'on a single training/testing example,' yet per-sample wall-clock times in milliseconds can be dominated by one-off overheads such as library import, JIT compilation, and graph construction. No accuracy or loss is reported for any task, so it is impossible to determine whether the implementations actually learned the tasks; a fast-but-failed run would still enter the table. For the only entries with standard deviations (MNIST Sum training), DeepProbLog at 5.84 +/- 3.24 ms and Scallop at 6.50 +/- 2.35 ms overlap substantially, so the conclusion that 'DeepProbLog achieves slightly faster training performance than Scallop' is not statistically supported. The claims of fastest inference and memory efficiency in Section 6 and the tradeoff discussion in Section 9 therefore rest on an inadequate empirical basis.","section":"Section 6, Table 2"},{"comment":"The four tasks are all small, synthetic examples (two-digit MNIST Sum, synthetic Shapes, randomly generated Toy NER embeddings, and random-number Math Equation Inference), and all implementations are provided by the authors. The manuscript does not report hardware, training epochs, optimization hyperparameters, number of trials beyond five runs, or any convergence criteria, making the numbers non-reproducible. Because the authors are also the developers of DomiKnowS, differential familiarity with one framework is an uncontrolled confound. The text itself attributes DomiKnowS's slower training to 'the overhead of uploading the entire graph of data into memory' and DeepProbLog-Scallop differences to 'overhead unrelated to the core algorithmic complexity,' which concedes that the measurements reflect implementation details rather than framework-level algorithmic properties. Without independent implementations or at least a comparison using the official repositories and a detailed experimental protocol, the claimed efficiency tradeoffs are not established.","section":"Section 8, Table 2"},{"comment":"The claim that 'Datalog can use top-k results and exploit database optimizations, making Scallop algorithmically more time-efficient than DeepProbLog' is presented as an established fact, but no algorithmic complexity analysis or reference is provided. This theoretical expectation is then used in Section 6 to interpret the empirical results, so it is a load-bearing part of the efficiency comparison. It should either be formally justified or explicitly hedged as a conjecture.","section":"Section 3"}],"minor_comments":[{"comment":"The layout of Table 1 is difficult to read: the 'Eff' column appears to contain citations (e.g., 'Faghihi et al. (2024)') and check/cross marks are placed in ambiguous columns; the table should be reformatted so that each facet column is clearly aligned for every framework.","section":"Table 1"},{"comment":"The abstract says the paper showcases three generic frameworks, but Section 8 also implements LEFT for the Simple VQA task; the paper should either present LEFT as a fourth showcased framework or clearly state that LEFT is an additional illustrative example only.","section":"Abstract and Section 8"},{"comment":"Section 7 discusses DomiKnowS's use of LLMs (referencing Prompt2DeModel), but Table 1 marks the LLM column for DomiKnowS as '✗'; this inconsistency should be resolved.","section":"Section 7 and Table 1"},{"comment":"The Math Equation Inference task states that each list contains six real numbers, but the property descriptions use 'P8 i=0 xi > 0' and 'P8 i=0 |xi| > 0.5', which suggests a sum over nine elements; the indexing should be corrected to match the list length.","section":"Section 8.4"},{"comment":"The paper would benefit from an explicit Limitations subsection that acknowledges the toy-scale tasks, the lack of independent implementations, and the potential for implementation artifacts in the timing measurements, as these are currently only implicit in the discussion.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The authors have a clear stake in one of the compared frameworks (DomiKnowS), and the empirical comparison is based entirely on their own implementations. For a journal-level comparative study, I would recommend requiring a substantially strengthened empirical section: either independent or officially sourced implementations, accuracy and convergence metrics, and a reproducible hardware/software protocol. The qualitative taxonomy is promising and worth preserving, but the current empirical claims need to be supported or explicitly downgraded to anecdotal illustrations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a genuinely useful conceptual map of three NeSy frameworks, with an empirical section that does not support the tradeoff claims the paper draws from it. The five-facet characterization—symbolic knowledge representation, neural modeling flexibility, model declaration, symbolic/sub-symbolic interplay, and LLM usage—is a solid way to organize DeepProbLog, Scallop, and DomiKnowS, and the code-level walkthroughs on four tasks give practitioners something concrete to react to. Table 1 is a handy compact summary. That part earns a serious read.\n\nThe stress-test note is right. Section 6 introduces Table 2 as a comparison of computational efficiency, but the text says the numbers come from 'a single training/testing example.' Per-sample milliseconds on toy tasks are dominated by one-off costs—library imports, compilation, graph construction, solver startup—so they cannot be read as framework-level properties. The table reports no accuracy, so a run that failed to learn would still show up. Where error bars appear, they overlap: DeepProbLog 5.84 ± 3.24 ms versus Scallop 6.50 ± 2.35 ms on MNIST Sum training does not support 'slightly faster training.' The authors mention overhead as a possible cause, but the caveat is too weak given how the table is used.\n\nThe DomiKnowS stake is a real bias risk. The authors built that framework and several related systems, and the qualitative sections consistently frame DomiKnowS in the most favorable light. That does not make the taxonomy circular, but those sections should have been written or checked by someone without a stake. The citation pattern is broad and mostly fine; self-citation is heavier than I'd like but not disqualifying.\n\nWho this is for: NeSy researchers and practitioners who want a structured starting point for choosing a framework or understanding what genericity means. Those readers get value from Sections 2–5 and 8. Nobody should choose a framework based on Table 2.\n\nMy recommendation: send to peer review, with the empirical comparison either substantially reworked—accuracy reported, multiple seeds and runs, benchmark-level tasks, proper handling of setup costs—or explicitly reframed as illustrative timings of specific reference implementations. The taxonomy is worth publishing; the timing table as presented is not yet evidence.","headline":"A useful conceptual taxonomy of NeSy frameworks, but the efficiency table is too thin to support the claimed tradeoffs.","tokens_in":21799,"tokens_out":3308,"would_cite":true,"duration_ms":31044,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generic neurosymbolic frameworks can be characterized by five technical facets, and a four-task benchmark shows measurable speed-memory tradeoffs among DeepProbLog, Scallop, and DomiKnowS.","keywords":["neurosymbolic AI","framework comparison","DeepProbLog","Scallop","DomiKnowS","declarative learning","differentiable reasoning","LLM-assisted symbolic knowledge"],"falsifier":"Run the same tasks with the same neural backbone, optimizer, batch size, and hardware, using implementations maintained independently for each framework; if DeepProbLog's training-speed advantage and Scallop's inference-speed advantage disappear or reverse, the reported tradeoffs are implementation artifacts rather than properties of the frameworks themselves.","tokens_in":20847,"feed_emoji":"🧠","tokens_out":10634,"duration_ms":84814,"temperature":0.7,"pith_summary":"Neurosymbolic AI aims to join neural learning with symbolic reasoning, but most work is algorithmic and task-specific rather than offering general-purpose tools. The paper's central claim is that the few generic frameworks that do exist can be understood through five facets: symbolic knowledge representation, neural modeling flexibility, model declaration, the interplay between symbolic and sub-symbolic systems, and use of large language models. It demonstrates the facets on DeepProbLog, Scallop, and DomiKnowS by implementing shared tasks (MNIST Sum, Shapes, Toy NER, Math Equation Inference, and a simple VQA) and measuring training time, inference time, and memory use. The empirical results show Scallop with the fastest inference, DeepProbLog training slightly faster in some settings, and DomiKnowS slower to train because it loads the whole domain graph into memory. A sympathetic reader would care because the facet characterization converts framework selection from a matter of taste into a structured engineering tradeoff and points to what the next generation of frameworks should fix.","feed_headline":"Five facets separate logic-neural frameworks; speed tests show tradeoffs","feed_subtitle":"A four-task benchmark of DeepProbLog, Scallop, and DomiKnowS shows each framework trades speed for flexibility.","key_machinery":"The carrying mechanism is the five-facet characterization summarized in Table 1 and then instantiated by shared tasks. Table 1 encodes each framework's symbolic language (ProbLog vs Datalog vs graph/FOL-like constraints), whether model declaration is flexible, which interplay algorithm is supported, computational-efficiency notes, and whether LLMs are used. The tasks act as a controlled stage on which the facets produce observable differences; Table 2 records per-sample training time, testing time, and memory for that purpose. The theoretical engine behind the efficiency differences is the choice of underlying reasoning formalism: exhaustive probabilistic inference over arithmetic circuits in DeepProbLog, top-k Datalog with provenance semirings in Scallop, and ILP-based constraint solving over an explicit domain graph in DomiKnowS.","core_discovery":"The paper's own claim is that differences among generic neurosymbolic frameworks are not incidental implementation details but consequences of design choices across five facets. DeepProbLog builds on ProbLog's exhaustive probabilistic semantics with arithmetic circuits; Scallop relaxes those semantics with Datalog-style top-k reasoning and provenance semirings in a Rust implementation; DomiKnowS treats domain knowledge as a graph with logical constraints solved by integer linear programming, Lagrangian primal-dual training, or sampling losses. The tabulated results report that Scallop achieves the fastest testing times, DeepProbLog is slightly faster to train than Scallop in several settings, and DomiKnowS training is the slowest because of graph-loading overhead. The same facet lens shows that current frameworks each support only a narrow slice of possible symbolic-neural interplays, which the paper argues should be broadened in future designs.","pith_inferences":["I infer that the precise time and memory ratios should be treated as hypotheses rather than settled facts, because they come from the authors' own implementations; a controlled reimplementation with matched neural backbones could confirm or overturn the framework-level ordering.","I infer that the five-facet rubric could be applied to newer frameworks to predict integration effort before benchmarking, for example by scoring each facet and correlating it with measured user task-completion time.","I infer a testable claim about algorithm versus engineering: if Scallop's Datalog-style top-k optimizations are the true source of its speed, replacing its Rust runtime with a slower interpreter should preserve a smaller but still measurable advantage over DeepProbLog's exhaustive circuits.","I infer that the LLM facet could be quantified by measuring user time to specify a new task in natural language versus hand-crafted rules, which would test the claimed advantage of LLM-assisted model declaration."],"forward_implications":["If the facet characterization holds, framework selection becomes a requirements analysis: tasks needing exact probabilistic semantics fit DeepProbLog, tasks needing scalable differentiable Datalog reasoning fit Scallop, and tasks needing per-concept distant supervision fit DomiKnowS.","If the benchmark numbers generalize, a deployment that runs many inference queries should favor Scallop's speed, while a training-heavy research setting might tolerate DomiKnowS's slower training in exchange for concept-level losses.","If the facet lens is adopted, future framework papers should report symbolic representation language, model-declaration flexibility, supported interplay algorithms, LLM integration, and per-sample time and memory, making comparisons reproducible.","If the identified gaps are real, next-generation frameworks should support multiple types of symbolic-neural interplay in one system and use LLMs to generate or refine symbolic knowledge, lowering the hand-crafting barrier."],"supporting_citations":[{"why":"Provides DeepProbLog, the probabilistic logic programming framework whose semantics and performance the paper compares.","marker":"Manhaeve et al. (2021)"},{"why":"Introduces Scallop's differentiable reasoning over Datalog, the central object of the comparison.","marker":"Huang et al. (2021)"},{"why":"Defines Scallop as a language for neurosymbolic programming, grounding the framework's relational data model.","marker":"Li et al. (2023b)"},{"why":"Shows Scallop-based relational programming with foundation models, used for the LLM facet and the VIERA example.","marker":"Li et al. (2024)"},{"why":"Introduces DomiKnowS, the graph and constraint based framework compared throughout the paper.","marker":"Rajaby Faghihi et al. (2021)"},{"why":"Supplies DomiKnowS's natural-language-to-constraints interface, cited for LLM-based model declaration.","marker":"Faghihi et al. (2024)"},{"why":"Provides ProbLog, the probabilistic extension of Prolog that DeepProbLog builds on.","marker":"De Raedt et al. (2007)"},{"why":"Offers the taxonomy of symbolic-subsymbolic interplay types used to compare frameworks.","marker":"Kautz (2022)"},{"why":"Gives the primal-dual constrained-optimization algorithm DomiKnowS supports.","marker":"Nandwani et al. (2019)"},{"why":"Supplies inference-masked loss, one of the loss formulations used in DomiKnowS's constraint solving.","marker":"Guo et al. (2020)"}],"fun_headline_variants":["Five facets define logic-neural framework tradeoffs","NeSy frameworks: speed vs flexibility across five facets","Comparing DeepProbLog, Scallop, DomiKnowS: design tradeoffs","Three generic NeSy frameworks, five facets, speed-flexibility tradeoff","Neural-symbolic frameworks compared: five facets decide speed vs expressivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the four toy tasks and the authors' implementations being representative of each framework's general expressivity, and on wall-clock time and memory differences reflecting framework-level algorithmic properties rather than uneven engineering effort.","fun_headline_variants_meta":{"raw":{"variants":["Five facets define logic-neural framework tradeoffs","NeSy frameworks: speed vs flexibility across five facets","Comparing DeepProbLog, Scallop, DomiKnowS: design tradeoffs","Three generic NeSy frameworks, five facets, speed-flexibility tradeoff","Neural-symbolic frameworks compared: five facets decide speed vs expressivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001046,"raw_usage":{"total_tokens":4385,"prompt_tokens":925,"completion_tokens":3460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":3369}},"tokens_in":541,"tokens_out":3460,"duration_ms":21246,"temperature":1.0,"reasoning_tokens":3369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:13:19.257053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same tasks with the same neural backbone, optimizer, batch size, and hardware, using implementations maintained independently for each framework; if DeepProbLog's training-speed advantage and Scallop's inference-speed advantage disappear or reverse, the reported tradeoffs are implementation artifacts rather than properties of the frameworks themselves.","supporting_citations":[{"cited_title":"Scallop: From probabilistic deductive databases to scalable differentiable reasoning","cited_arxiv_id":null,"evidence_quote":"Introduces Scallop's differentiable reasoning over Datalog, the central object of the comparison."},{"cited_title":"D omi K now S : A library for integration of symbolic domain knowledge in deep learning","cited_arxiv_id":null,"evidence_quote":"Introduces DomiKnowS, the graph and constraint based framework compared throughout the paper."},{"cited_title":"Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language","cited_arxiv_id":"2407.20513","evidence_quote":"Supplies DomiKnowS's natural-language-to-constraints interface, cited for LLM-based model declaration."},{"cited_title":"A primal dual formulation for deep learning with constraints","cited_arxiv_id":null,"evidence_quote":"Gives the primal-dual constrained-optimization algorithm DomiKnowS supports."},{"cited_title":"Inference-masked loss for deep structured output learning","cited_arxiv_id":null,"evidence_quote":"Supplies inference-masked loss, one of the loss formulations used in DomiKnowS's constraint solving."}],"review_version":2}