Enhancing Computer Vision with Knowledge: a Rummikub Case Study

Joost Vennekens; Laurent Mertens; Simon Vandevelde; Sverre Lauwers

REVIEW 3 major objections 4 minor 15 references

Enhancing Computer Vision with Knowledge: a Rummikub Case Study

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A logical correction step can substitute for about two-thirds of a vision system's training data.

desk verdict A transparent Rummikub case study showing a neurosymbolic pipeline can cut data needs—but the headline claim is measured in-sample, so the actual savings remain unproven until a held-out evaluation appears. read the letter →

arxiv 2411.18172 v1 pith:DUJFFH2I submitted 2024-11-27 cs.CV cs.LO

Simon Vandevelde , Laurent Mertens , Sverre Lauwers , Joost Vennekens This is my paper · ORCID

classification cs.CVcs.LO

keywords Rummikubneurosymbolicreasoningknowledge-basedvisionobjectdetectionconstraintoptimizationdataefficiencylogicalcorrectionconfidence-basedclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Artificial neural networks are good at labeling individual Rummikub tiles but not at reconciling those labels with the game's rules. This paper tests whether an explicit, declarative model of the rules, used as a post-processing correction step, can make up for less data and less training. On their 285-image Rummikub dataset, the answer is yes: the knowledge-enhanced pipeline reaches its best corrected accuracy around 94.98% with 30% of the data, while the plain ANN needs 95% of the data to reach 94.79%. The same pattern holds for training time: 5 epochs with reasoning beats 20 epochs without. The point is that in structured visual tasks where output classes are constrained by knowable rules, background knowledge can be a direct substitute for data.

What carries the argument

The machinery is a constraint-optimization correction step run by the IDP-Z3 reasoning engine, a solver that finds optimal models of a first-order-logic knowledge base. After per-tile classifiers emit confidence scores for every possible number and color, the engine treats each clustered set of tiles as a decision problem: choose one number and one color per tile to maximize the sum of the confidence scores, subject to the formalized Rummikub rules. A group must have the same number with distinct colors, and a run must have the same color with consecutive numbers, with jokers allowed as wildcards. This is what changes per-tile greedy classification into globally consistent classification. The knowledge base encodes the rules once and is reused for the optimization, and the hand-crafted clustering step supplies the grouping that the optimization operates on.

What would settle it

Train the identical pipeline on a disjoint held-out test set of Rummikub photos and compare the knowledge-corrected model trained on 30% of the data with the pure ANN trained on 95%: if the corrected model no longer reaches the pure ANN's accuracy, the reported data-equivalence depends on testing on training images.

Watch

Extended reading notes

Core claim

The paper's central claim is that a correction step based on explicit logical knowledge can be as valuable as a large share of the training data. The pipeline detects tiles with an SSD detector, clusters them into candidate sets, and classifies each tile's number and color with ResNet18 networks that output confidence scores rather than hard labels. A reasoning engine then treats each candidate set as a constrained optimization problem: find the assignment of numbers and colors to tiles that maximizes the sum of confidence scores while satisfying the Rummikub rules for groups and runs, including jokers as wildcards. Across ten runs, this full pipeline at 30% of the data (94.98±1.79%) matches the plain ANN at 95% of the data (94.79±1.85%), and with 5 epochs (95.64±1.84%) matches what the ANN achieves with 20 (95.56±1.66%). The full pipeline also exceeds the best pure-ANN accuracy, and its standard deviation is consistently smaller, so the reasoning step acts as both an accuracy boost and a variance stabilizer.

Load-bearing premise

The headline comparison assumes that accuracy measured on the same images used for training is a fair proxy for how the models would perform on new, unseen game states; the paper explicitly uses the training set for testing.

Editorial extensions

If this is right

For the Rummikub scenario, a vision system can reach its best classification accuracy with 30% of the training data when a logical correction step is added, versus 95% without it.
The knowledge-enhanced pipeline reaches its best accuracy after 5 training epochs, while the pure ANN needs 20, cutting required training time by about three-quarters in the authors' experiments.
The correction step reduces run-to-run variance: between 30% and 45% of the data, classifier standard deviation stayed at 1.37–2.8 with reasoning versus 5.57–8.5 without.
Full pipeline accuracy is higher than the pure ANN's best: 98.76% at 90% data versus 94.79% at 95% in the data experiment, and 98.84% at 13 epochs versus 95.56% at 20 in the epoch experiment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If this data-versus-knowledge equivalence holds beyond Rummikub, then data-collection budgets in structured visual tasks such as form processing, industrial inspection, and sensor fusion could be traded against writing down domain rules once.
A clean next test is to hold out genuinely unseen Rummikub photos; because the paper's evaluation reuses the training images for testing, the exact size of the knowledge advantage on novel inputs is not yet established.
The same correction scheme should transfer to any vision task whose output labels must satisfy known combinatorial constraints, such as Sudoku, chess positions, or card games, where groupings may be less rigid than Rummikub's sets.
The pipeline could be extended from enforcing validity to detecting invalid sets and proposing repairs, turning the reasoning step into a source of explanations rather than only corrected labels.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's the short version: this is a clean, honestly written case study that transplants Mulamba et al.'s hybrid classification+reasoning pipeline from Sudoku to Rummikub, where tile positions aren't known, and for the first time quantifies how much data and training time the reasoning step saves. The numbers look good, but the headline equivalence—knowledge worth two-thirds of the data—is measured entirely on the training set, because the authors deliberately use the full dataset as test data (footnote 5). That makes the quantitative claim conditional until held-out results appear.

What's genuinely good: the writing is crisp; the pipeline is transparent; code (GitLab) and data (Kaggle) are public; the authors admit that train=test is bad practice and clearly flag their assumption. The variance reduction from adding the reasoning step is a nice observation, and the measured inference overhead (0.1s per set) is honest and useful. The related work is fair, crediting Mulamba et al. directly.

The soft spots are in the evaluation. In-sample accuracy can ride on dataset-specific cues like background, lighting, and tile placement, so the 'two-thirds of data' figure is not yet shown to transfer to new images. The authors argue that the setup gives the pure ANN 'ideal circumstances,' but even if that's true for the ANN, it doesn't validate the equivalence for unseen data. A held-out split (or a separate test set) would settle it. Minor: the clustering step is hand-crafted and underdescribed, so exact reproduction needs more details. The correction step assumes the game state is valid; that's fine for the stated use case, but the conclusion section is a bit optimistic about generalizing to forms and sensor fusion without addressing how validity assumptions are obtained there.

The circularity concern from the stress test does not land: the IDP-Z3 step is a legitimate constraint optimization over the Rummikub rules, not a circular derivation.

Bottom line: this paper deserves a serious referee, but the central quantitative claim needs out-of-sample evidence before I'd rely on it. I'd suggest sending it to review with a request for a held-out evaluation and a fuller description of the clustering hyperparameters. For reading group, it's a good short discussion piece on the gap between in-sample and real-world gains in neurosymbolic systems.

Referee Report

3 major / 4 minor

Summary. The paper presents a four-stage pipeline for detecting and classifying Rummikub tiles in images: an SSD detector for bounding boxes, a hand-crafted clustering algorithm that groups boxes into candidate sets, two ResNet18 networks for number and color classification, and an IDP-Z3 optimization step that re-ranks the neural confidences under the constraint that every detected set is a valid Rummikub group or run. The central empirical claim is that adding this logical correction step is worth roughly two-thirds of the training data and about half the training epochs, based on ablation experiments in Section 3 that vary the amount of data (Fig. 3a) and the number of training epochs (Fig. 3b).

Significance. If the result survives a held-out evaluation, it is a valuable demonstration that declarative knowledge and constraint-based reasoning can substitute for a substantial fraction of labeled data in a structured visual domain. The paper's strengths include a publicly available dataset, a GitLab repository with code, a clean formulation of the Rummikub rules in IDP-Z3, and an ablation design that directly quantifies the contribution of the reasoning step. The main weakness is that the headline equivalence is measured on the training set and appears to be within statistical noise.

major comments (3)

[Section 3, footnote 5] The evaluation uses the full dataset as test data while subsets of the same images are used for training. This is acknowledged in footnote 5 as 'considered bad practice'. Because both curves in Fig. 3a are in-sample accuracy, the central claim that background knowledge is 'equally valuable as two-thirds of the data set' is not established for unseen images. The pure ANN can exploit dataset-specific cues, and the reported data-efficiency gain may not transfer to new game states. Please add a held-out evaluation, for example a train/test split or repeated cross-validation on the 285 images, and report whether the 30%-plus-reasoning versus 95%-pure-ANN equivalence persists.
[Section 3, Fig. 3a] The claimed equivalence between knowledge and data rests on two values that are statistically indistinguishable: IDP-Z3 at 30% of the data reaches 94.98±1.79% and the pure ANN at 95% reaches 94.79±1.85%. The difference is about 0.2 percentage points, well within one standard deviation of the 10 runs. The paper should provide confidence intervals or a paired significance test to justify the 'two-thirds of the data' claim; without this, the headline number may be due to noise.
[Section 3, Fig. 3b] The same statistical caveat applies to the training-time claim. The paper states that 5 epochs with IDP-Z3 are sufficient instead of 20, but the reported values are 95.64±1.84% and 95.56±1.66%, respectively, which overlap within one standard deviation. A significance test or effect-size measure is needed to support the conclusion that reasoning halves the required training time.

minor comments (4)

[Section 2, Step 4] There is a typo in 'explicitely' in the description of Step 4; it should be 'explicitly'.
[Section 5, Conclusion] The phrase 'constraint by hardware limitations' should read 'constrained by hardware limitations'.
[Section 2, example confidences] In the worked example, Tile 3 is listed as '(3, red) → 0.5, (2, blue) → 0.15, (2, black) → 0.05, (2, orange) → 0.3'; the numbers after the colors appear to be typos and should probably be (3, blue), (3, black), and (3, orange) for consistency with the narrative.
[Section 3, first paragraph] The sentence 'the ANNs had a standard deviation between 5.57 and 8.5' mixes singular and plural; it would read more clearly as 'the ANN results had standard deviations between 5.57 and 8.5'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison whose central claim is measured, not derived from its own assumptions.

full rationale

The derivation chain in this paper is an experimental pipeline comparison, not a formal derivation that reduces to its inputs. The correction step in Section 2 (Step 4) uses Rummikub's game rules—groups and runs—as declared constraints in an optimization problem, and the paper does not claim to predict those rules from data. The central quantitative claim, that background knowledge is worth about two-thirds of the dataset, is an observed equivalence between two measured accuracy curves in Fig. 3a: IDP-Z3 at 30% of the data reaches 94.98% while the pure ANN reaches 94.79% at 95%. This is a comparison of empirical results, not a quantity defined in terms of a fitted constant or an equation that equals its own input. The footnote about using the full dataset as test data is a real validity concern, but it is an evaluation-protocol issue, not a circularity: the same dataset being used for training and testing does not make any result true by construction in the sense of self-definition or renamed fitting. The self-citations to IDP-Z3 and to the authors' own knowledge-base paradigm are tool and background citations; they are not invoked as an unverified uniqueness theorem or as the justification for the measured data-efficiency claim. The reasoning step is a genuine constraint-based correction of neural confidences, and the paper's contribution is an empirical ablation rather than a derivation that presumes its conclusion. Therefore, no circular step can be exhibited with a specific reduction, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions: the validity of the game state, the correctness of the encoded rules, and the ground truth of annotations. The only hand-chosen free parameter we can identify is the clustering hyperparameter set, which is not disclosed. No invented entities are introduced.

free parameters (1)

Clustering algorithm hyperparameters = Not reported
Step 2 is described as a hand-crafted algorithm 'designed to be robust to size and orientation'. Its distance thresholds or grouping rules are not specified, and they influence which tiles are grouped into sets before correction.

assumptions (3)

domain assumption The depicted Rummikub game state consists entirely of valid sets
Step 4 explicitly assumes the shown game state to be valid, which is what makes the correction optimization feasible. If the image can contain invalid sets, the correction would force an invalid state onto a valid explanation.
domain assumption The Rummikub rules encoded in the knowledge base are correct and complete for the task
Section 2 formalizes groups and runs; the correction step depends on these rules being the right constraints. The paper gives the example formula for groups, and the runs rule is mentioned but not fully formalized.
domain assumption The annotation of the dataset is ground truth
The paper relies on manually annotated bounding boxes and labels for training and evaluation; any annotation errors propagate into the measured accuracies.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Computer Vision with Knowledge: a Rummikub Case Study." pith.science (2026). https://pith.science/paper/DUJFFH2I

@misc{pith2026241118172,
  author       = {Pith},
  title        = {Pith review of: Enhancing Computer Vision with Knowledge: a Rummikub Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUJFFH2I}},
  note         = {Machine review of arXiv:2411.18172}
}

read the original abstract

Artificial Neural Networks excel at identifying individual components in an image. However, out-of-the-box, they do not manage to correctly integrate and interpret these components as a whole. One way to alleviate this weakness is to expand the network with explicit knowledge and a separate reasoning component. In this paper, we evaluate an approach to this end, applied to the solving of the popular board game Rummikub. We demonstrate that, for this particular example, the added background knowledge is equally valuable as two-thirds of the data set, and allows to bring down the training time to half the original time.

Figures

Figures reproduced from arXiv: 2411.18172 by the authors.

**Figure 1.** Rummikub set examples IDP-Z3 is a logical reasoning engine for first order logic. It adheres to the Knowledge Base Paradigm [4], which states that knowledge should be modeled declaratively in a Knowledge Base (KB) regardless of its use. As such, the KB is merely a “bag of knowledge”.To put this knowledge to use, it can be given to a reasoning engine such as IDP-Z3, which supports several kinds of inference tasks. Th… view at source ↗

**Figure 2.** Overview of the tile detection/classification pipeline [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Ablation experiment results. The dashed red lines indicate the highest [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

FindingEmo : An image dataset for emotion recognition in the wild (accepted at NeurIPS 2024)

Laurent Mertens, Elahe' Yargholi, Hans Op de Beeck , Jan Van den Stock , and Joost Vennekens. FindingEmo : An image dataset for emotion recognition in the wild (accepted at NeurIPS 2024). 2024

work page 2024
[2]

Context modeling in computer vision: Techniques, implications, and applications

Oge Marques, Elan Barenholtz, and Vincent Charvillat. Context modeling in computer vision: Techniques, implications, and applications. Multimedia Tools and Applications , 51(1):303--339, January 2011

work page 2011
[3]

IDP-Z3 : A reasoning engine for FO (.)

Pierre Carbonnelle, Simon Vandevelde, Joost Vennekens, and Marc Denecker. IDP-Z3 : A reasoning engine for FO (.). arXiv preprint arXiv:2202.00343 , 2022

arXiv 2022
[4]

Building a Knowledge Base System for an Integration of Logic Programming and Classical Logic

Marc Denecker and Joost Vennekens. Building a Knowledge Base System for an Integration of Logic Programming and Classical Logic . In Logic Programming , volume 5366, pages 71--76. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008

work page 2008
[5]

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Computer Vision -- ECCV 2016 , pages 21--37, Cham, 2016. Springer International Publishing

work page 2016
[6]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016

2016
[7]

Deepproblog: Neural probabilistic logic programming

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems , 31, 2018

2018
[8]

Neurasp: Embracing neural networks into answer set programming

Zhun Yang, Adam Ishay, and Joohyung Lee. Neurasp: Embracing neural networks into answer set programming. In Proceedings of IJCAI-20 , pages 1755--1762. International Joint Conferences on Artificial Intelligence Organization, July 2020

work page 2020

Show all 15 references

[9]

From statistical relational to neurosymbolic artificial intelligence: A survey

Giuseppe Marra, Sebastijan Duman c i \'c , Robin Manhaeve, and Luc De Raedt. From statistical relational to neurosymbolic artificial intelligence: A survey. Artificial Intelligence , 328:104062, March 2024

2024
[10]

Hybrid Classification and Reasoning for Image-Based Constraint Solving

Maxime Mulamba, Jayanta Mandi, Rocsildes Canoy, and Tias Guns. Hybrid Classification and Reasoning for Image-Based Constraint Solving . In Integration of Constraint Programming , Artificial Intelligence , and Operations Research , pages 364--380, 2020

2020
[11]

Haykin, editor

S. Haykin, editor. Unsupervised Adaptive Filtering vol.1 : Blind Source Separation, John Willey ans Sons, New York, 2000

2000
[12]

Delfosse and P

N. Delfosse and P. Loubaton, Adaptibe blind separation of sources: A deflation
[13]

Cruces, A

S. Cruces, A. Cichocki and S. Amari, The minimum entropy and cumulants based contrast functions for blind source extraction. In J. Mira and A. Prieto, editors, proceedings of the 6 ^ th international workshop on artificial neural networks ( IWANN 2001), Lecture Notes in Comput...

2001
[14]

Vrins, C

F. Vrins, C. Archambeau and M. Verleysen, Towards a local separation performances estimator using common ICA contrast functions? In M. Verleysen, editor, proceedings of the 12^ th
[15]

J. V. Stone and J. Porrill, Undercomplete independent component analysis for signal separation and dimension

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

FindingEmo : An image dataset for emotion recognition in the wild (accepted at NeurIPS 2024)

Laurent Mertens, Elahe' Yargholi, Hans Op de Beeck , Jan Van den Stock , and Joost Vennekens. FindingEmo : An image dataset for emotion recognition in the wild (accepted at NeurIPS 2024). 2024

work page 2024

[2] [2]

Context modeling in computer vision: Techniques, implications, and applications

Oge Marques, Elan Barenholtz, and Vincent Charvillat. Context modeling in computer vision: Techniques, implications, and applications. Multimedia Tools and Applications , 51(1):303--339, January 2011

work page 2011

[3] [3]

IDP-Z3 : A reasoning engine for FO (.)

Pierre Carbonnelle, Simon Vandevelde, Joost Vennekens, and Marc Denecker. IDP-Z3 : A reasoning engine for FO (.). arXiv preprint arXiv:2202.00343 , 2022

arXiv 2022

[4] [4]

Building a Knowledge Base System for an Integration of Logic Programming and Classical Logic

Marc Denecker and Joost Vennekens. Building a Knowledge Base System for an Integration of Logic Programming and Classical Logic . In Logic Programming , volume 5366, pages 71--76. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008

work page 2008

[5] [5]

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Computer Vision -- ECCV 2016 , pages 21--37, Cham, 2016. Springer International Publishing

work page 2016

[6] [6]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016

2016

[7] [7]

Deepproblog: Neural probabilistic logic programming

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems , 31, 2018

2018

[8] [8]

Neurasp: Embracing neural networks into answer set programming

Zhun Yang, Adam Ishay, and Joohyung Lee. Neurasp: Embracing neural networks into answer set programming. In Proceedings of IJCAI-20 , pages 1755--1762. International Joint Conferences on Artificial Intelligence Organization, July 2020

work page 2020