{"id":"e0ef58a0-2c68-484d-9093-ce5f1b749036","arxiv_id":"1908.01931","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Standard neural networks learn bitwise and simple arithmetic from digit images but fail at multiplication; a divide-and-conquer model with carry/non-carry labels reaches 84.5% on multiplication.","lead":"This paper tests whether neural networks can infer arithmetic and bitwise logic directly from images of digits, without being told the operation. It finds that standard models master easy and intermediate operations, but multiplication requires a hand-designed decomposition into carry and non-carry parts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hardest empirical result (84.5% on multiplication) is obtained by the DCM after the authors insert human-specified carry/non-carry labels, so the claim that logic is learned 'without providing any reasoning patterns beforehand' is not supported for the difficult task.","rationale":"The reader selected OCR reliability as the weakest assumption. That concern is legitimate but does not strike at the central claim: the paper itself acknowledges that OCR can reject correct digits, so all reported accuracies are lower bounds, and the qualitative pattern (easy tasks learnable, multiplication fails for standard models, DCM helps) would survive even a downward correction. The more load-bearing issue is that the headline claim—'logic could be learned from images without presetting any reasoning patterns'—is not tested on the hardest task in its clean form. The DCM is trained with explicit carry and non-carry ground-truth labels, which are a human-specified reasoning pattern. Without those labels, multiplication accuracy is near zero. This does not make the paper worthless: it still contributes a new visual reasoning benchmark and shows that several architectures can learn bitwise, addition, and subtraction from pixels, and it demonstrates a useful divide-and-conquer engineering remedy. But it means the central conceptual claim is overbroad, and this should be stated clearly in any accepted version. The reader's conditional verdict already captures this in its rationale, so my concern does not change the verdict; it sharpens the reason for the condition: the DCM result must be presented as combining human-specified structure with data-driven learning, not as evidence that complex logic can be learned with no preset reasoning patterns.","tokens_in":20114,"tokens_out":5536,"duration_ms":57990,"concrete_test":"Train the DCM architecture end-to-end on the original two-image multiplication task, removing the human-provided C and D labels (for example, by replacing the explicit carry/non-carry subtasks with latent intermediate representations learned from data, or by directly predicting the final output image e from inputs a and b using the same three-module capacity). Evaluate on the same test split as in Table 5. If accuracy remains near 84.5%, the DCM result does not depend on the prespecified carry/non-carry decomposition; if accuracy falls to the roughly 0.2% level of the MLP/CNN baselines, the reported success is contingent on the human-specified reasoning pattern and the 'without presetting' claim must be restricted to the easier tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a LiLi task can be learned 'without providing any reasoning patterns beforehand' (Section 3.2). The strongest empirical evidence for the difficult multiplication task, however, comes from the DCM, which is trained on a regenerated dataset containing extra images C and D that explicitly encode the carry and non-carry parts of multiplication (Section 5, Fig.13). These labels are a human-specified decomposition of the arithmetic reasoning pattern; the model does not discover the carry/non-carry structure from the original two-image input. On the original multiplication images, the best typical network reaches only about 0.35% accuracy (Table 3), while the DCM reaches 84.5% using the human-provided intermediate targets. The conclusion also concedes that the DCM works by 'combining domain expert knowledge with data-driven model.' Thus the DCM's success supports divide-and-conquer with preset reasoning structure, not the unassisted 'logic could be learned from images' claim. The OCR issue raised by the reader is real but secondary: the paper explicitly notes OCR can fail on correct digits, so the reported accuracies are lower bounds, and the easy-task conclusions (bitwise, addition, subtraction) remain qualitatively robust even if OCR error is non-negligible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a 'data concept logic' (DCL) and a 'learning logic from images' (LiLi) task in which a model receives two rendered images of numbers and must output the rendered image of the result of an unknown binary operation. Six datasets are constructed: bitwise AND/OR/XOR, addition, subtraction, and multiplication. The authors train CNN-LSTM, MLP, CNN-MLP, autoencoder, and ResNet variants; with 150,000 training samples these models reach near-100% accuracy on the bitwise tasks and roughly 99% on addition/subtraction, while all remain below 0.4% on multiplication. To address multiplication, the paper proposes a divide-and-conquer model (DCM) that is trained with additional human-defined carry and non-carry images and reports 84.5% accuracy. The paper concludes that logic patterns can be learned from images without preset reasoning patterns, while also stating that the DCM combines domain expert knowledge with data-driven models.","tokens_in":20380,"tokens_out":6473,"duration_ms":66863,"significance":"The LiLi benchmark is a useful diagnostic: it has a clear single-answer evaluation, a controlled difficulty gradient, and the experiments show that ordinary architectures can learn arithmetic from raw pixel pairs surprisingly well. The consistently high accuracies on bitwise and addition/subtraction tasks across several architectures are credible and constitute the paper's main empirical contribution. The complete failure of all tested networks on multiplication, and the substantial improvement from a hand-decomposed training target, are informative for the community even though the decomposition is provided by the authors. The paper does not provide code or data links, and no formal or machine-checked proofs are included, so reproducibility rests on the textual dataset description and hyperparameter tables.","major_comments":[{"comment":"The claim in Section 3.2, after Eq. (9), that 'the LPN merely needs to be provided some training data to automatically learn the logical patterns... without providing any reasoning patterns beforehand' is not supported for the hard task. In Section 5 and Fig. 13, the DCM is trained on a regenerated multiplication dataset in which the carry image C and non-carry image D are human-defined intermediate targets; this is exactly a preset decomposition of the multiplication reasoning pattern. The paper's own conclusion states that the DCM works by 'combing the domain expert knowledge with data-driven model.' The central claim should be revised to apply only to the easy and intermediate tasks, or the DCM should be presented as a hybrid method rather than evidence for unsupervised logic discovery.","section":"Section 3.2 / Section 5"},{"comment":"Accuracy is measured by Tesseract OCR on predicted images, and the authors note that 'Some are right but OCR can not recognize them.' Because no OCR accuracy on the ground-truth images is reported, the percentages in Tables 2-5 are uncalibrated with respect to actual image-level correctness. The 100% easy-task results are robust to this concern, but the DCM's 84.5% and the multiplication failure rates could shift if OCR errors are not symmetric; an OCR baseline on ground-truth outputs or a pixel/character-level metric should be provided.","section":"Section 4.2"},{"comment":"The relationship between the reported 84.5% DCM accuracy and the per-subtask accuracies (carry 86.25%, non-carry 98.38%, synthetic 84.46%) is unclear. The testing procedure passes predicted carry/non-carry images c' and d' into the synthetic subtask, so errors in the first two subtasks propagate; if the 84.5% is instead the synthetic subtask accuracy evaluated with ground-truth c and d, then the end-to-end accuracy on the original a,b inputs is not reported. The authors should state exactly which pipeline is evaluated and report end-to-end accuracy.","section":"Table 5 / Fig. 14"}],"minor_comments":[{"comment":"Formulas 5-7 contain repeated typos, including 'squence' for 'sequence' and 'T he' for 'The.'","section":"Section 2.2"},{"comment":"The text contains 'CNN-NLP' where 'CNN-MLP' is clearly intended, and 'verity' should be 'verify.'","section":"Section 4.2"},{"comment":"Eq. (14) is not operationalized: no definition or proxy for 'difficulty' is given, so the inequality H > f(h1,...,hk) reads as a heuristic rather than a testable criterion.","section":"Section 5"},{"comment":"Table 1 does not include learning rates, optimizers, or epoch counts for all models, despite the text saying that the hyper-parameter settings are shown in Table 1; adding these details would improve reproducibility.","section":"Section 4.1"},{"comment":"The relationship between the LiLi task and the Fashion-Logic dataset of reference [23] should be stated explicitly, since both are described as data-driven visual logic tasks.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable as an empirical study if the claims are scoped; the current title and abstract promise more than the DCM experiments deliver, and the reported accuracies need an OCR calibration caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things: the LiLi dataset design and the systematic failure of standard models on multiplication are real contributions; the 84.5% DCM result is not evidence for the headline claim. The six image-arithmetic tasks are clearly specified, and the easy/intermediate levels are convincingly learned (near 100% for bitwise, ~99% for addition/subtraction) across several architectures at 150k training samples. The failure of generic convnets and MLPs on multiplication is a useful, reproducible-in-spirit empirical finding.\n\nWhat is genuinely new: the controlled benchmark, the coarse difficulty hierarchy, and the demonstration that typical architectures do not learn multi-digit multiplication from raw images. The DCM is a legitimate divide-and-conquer engineering idea, and the authors are transparent that it works by adding carry/non-carry label images during training. But that transparency is precisely why the paper's central claim—logic can be learned 'without providing any reasoning patterns beforehand' (Section 3.2)—does not hold for the hard task. The carry/non-carry decomposition is a human-specified reasoning pattern, and the conclusion concedes as much when it frames the model as combining domain expertise with data-driven learning.\n\nSoft spots, in proportion. The most important is the overclaim above; it needs reframing. Second, accuracy is measured with Tesseract OCR on synthetic output images, and the paper itself notes that correct digits can fail OCR. There is no ground-truth OCR baseline, so the reported numbers are lower bounds. The easy/intermediate conclusions are robust to this, but the 84.5% DCM figure could shift with a better reader. Third, there is no code or dataset release, which makes independent verification harder. Fourth, Hoshen and Peleg's earlier visual arithmetic learning is cited but not benchmarked against; that is a missed comparison, though not fatal. The formal DCL/inference section is mostly framing and adds little beyond supervised learning terminology.\n\nWho this is for: people building or evaluating visual arithmetic/reasoning benchmarks, and anyone studying whether CNNs can learn compositional algorithms from raw pixels. The math is simple and sound; the data generation is described well enough; the citation pattern is acceptable apart from the missing baseline.\n\nIt deserves a serious referee. I would send it to review with a clear request for major revision: soften the no-preset-patterns claim (at least for DCM), add an OCR accuracy check on ground-truth images, and release the data and code. After that, it would be a useful subfield contribution.","headline":"The datasets and negative results on multiplication are the real contribution; the DCM's 84.5% success relies on human-supplied carry labels, so the 'no preset reasoning patterns' claim is overreach.","tokens_in":20875,"tokens_out":3899,"would_cite":true,"duration_ms":99475,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["00-01","99-00"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that logical operations over numbers can be learned directly from images of those numbers, with no reasoning pattern supplied in advance, and that a divide-and-conquer network using extra label images lifts multiplication…","keywords":["logic reasoning","data concept logic","LiLi task","logical pattern network","divide and conquer model","arithmetic from images","bitwise operations","neural networks"],"falsifier":"Regrade the same test predictions by exact pixel comparison with ground-truth images, and also measure the OCR software's accuracy on clean ground-truth output images; if exact-pixel accuracy differs substantially from the OCR-based percentages, or if the OCR reader misreads clean ground-truth digits, then the reported results are artifacts of the recognition step rather than of logic learning.","tokens_in":19929,"feed_emoji":"🧮","tokens_out":8197,"duration_ms":78096,"temperature":0.7,"pith_summary":"This paper introduces a task it calls learning logic from images (LiLi): a network sees two images, each containing an integer rendered as pixels, and must produce an output image containing the result of an operation it was never told about. The authors report that standard networks reach about 100 percent accuracy on bitwise AND, OR, and XOR, and about 99 percent on addition and subtraction once the training set has 150,000 samples. Multiplication stays near chance for every standard model, so the paper proposes a divide-and-conquer model that splits the product into carry and non-carry parts using extra label images; that model reaches 84.5 percent test accuracy. The paper's position is that logical relations over numbers can be mined from pixel data alone, and that decomposition is a workable remedy when end-to-end learning fails.","feed_headline":"Neural networks learn arithmetic logic directly from images","feed_subtitle":"With 150,000 training images, standard nets master addition and bitwise ops; a split model cracks multiplication.","key_machinery":"The load-bearing objects are the data concept logic, defined as a triple $(I,R,O)$ in which $R$ is an unknown relation from input sequences to outputs; the LiLi task, which instantiates $I$ as pairs of images and $O$ as a result image; the Logical Pattern Network (LPN), a neural network trained by mean-square error that is supposed to absorb $R$ from data; and the divide-and-conquer model (DCM), an MLP-based pipeline that splits multiplication into carry, non-carry, and synthetic subtasks. The DCM carries the argument for the hard case by showing that a hand-chosen decomposition with intermediate label images reduces the uncertainty of each predicted digit and lifts multiplication accuracy from below one percent to 84.5 percent.","core_discovery":"The central claim is that a logic relation can be treated as a high-dimensional mapping $R:[-1,1]^{2K}\\to\\{0,1\\}^K$ between pixel values and learned by a neural network called the Logical Pattern Network, with no reasoning pattern specified in advance. On six synthetic image datasets, the paper reports that this succeeds almost perfectly for bitwise operations, succeeds for addition and subtraction once training data reaches 150,000 samples, and fails for multiplication with every tested standard architecture. The paper's explanation is that each output digit of a product depends on all preceding digit positions, so the uncertainty per digit is far larger than for addition or subtraction. The divide-and-conquer model attacks this by training separate MLP branches to predict the carry part and the non-carry part of the product, then a synthetic branch to combine them, reducing per-digit uncertainty from roughly 900 possibilities to 90 and yielding 84.5 percent test accuracy as the paper measures it.","pith_inferences":["If the central claim is right, the LiLi task could serve as a cleaner probe of machine reasoning than question-answering style benchmarks, because the only way to answer is to actually compute the relation from the two input images.","The OCR-based accuracy measure probably makes the reported numbers conservative: the paper states that some correct predicted digits are not recognized by the OCR software, so exact pixel-level grading might raise the reported accuracies.","The DCM's carry/non-carry decomposition is a human-specified reasoning pattern, which sits in tension with the paper's 'no reasoning patterns beforehand' slogan; the pure data-driven claim is really established only for the base tasks, while the hard-task success comes from injecting domain knowledge.","A testable extension is whether a sufficiently scaled end-to-end model, or an architecture with explicit positional alignment, can eventually match the DCM without decomposition; the paper's data-scaling results for addition suggest the gap may shrink with data."],"forward_implications":["If the results hold, bitwise logic operations can be acquired from raw pixels with no symbolic supervision: all tested models reach about 100 percent accuracy even with only 10,000 training samples.","Addition and subtraction, which require tracking a carry or borrow between adjacent digit positions, become learnable by CNN-MLP, Autoencoder, and ResNet models once the training set grows to 150,000 samples, supporting the paper's data-scaling strategy for hard logic.","Multiplication marks a sharp boundary for end-to-end neural networks: every standard model stays below 0.35 percent accuracy even at 150,000 samples, so the LiLi task isolates a genuinely hard reasoning step.","The DCM result implies that decomposing a complex visual logic task into subtasks, with extra images as labels, turns an unsolved problem into one that is mostly solvable: 84.5 percent test accuracy.","The same decomposition idea can be carried to other complex logic tasks, such as computing decimal bit operations by first converting to binary, as the paper's conclusion suggests."],"supporting_citations":[{"why":"Prior work on mining logical patterns from visual data, the immediate precursor to the LiLi task.","marker":"[23]"},{"why":"Existing abductive learning framework that combines perception and reasoning, which the paper contrasts with its fully data-driven approach.","marker":"[24]"},{"why":"Defines the ResNet architectures used as LPN baselines on all six datasets.","marker":"[26]"},{"why":"Universal approximation theorem used to justify that a neural network can represent the logical relation R.","marker":"[40]"},{"why":"Supplies the LSTM module used in the CNN-LSTM baseline.","marker":"[43]"},{"why":"Earlier visual arithmetic learning result that provides the MLP baseline and the task framing.","marker":"[44]"},{"why":"Autoencoder architecture used as an LPN baseline.","marker":"[46]"},{"why":"OCR software used to read digits from predicted images, defining the accuracy metric for every reported percentage.","marker":"[47]"}],"fun_headline_variants":["Nets learn logic from images, no preset patterns","Image-trained nets master bitwise ops, fail at multiplication","Divide-and-conquer nets crack image-based multiplication","Pixels teach neural nets arithmetic, but multiplication stumps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported accuracy depends on the OCR software's ability to read digits from the predicted images; the paper itself notes that at least some correctly predicted digits are not recognized by the OCR, so if OCR errors are non-negligible, both the failures of standard models and the 84.5 percent success of the DCM could be misstated.","fun_headline_variants_meta":{"raw":{"variants":["Nets learn logic from images, no preset patterns","Image-trained nets master bitwise ops, fail at multiplication","Divide-and-conquer nets crack image-based multiplication","Pixels teach neural nets arithmetic, but multiplication stumps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3503,"prompt_tokens":952,"completion_tokens":2551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2487}},"tokens_in":568,"tokens_out":2551,"duration_ms":18681,"temperature":1.0,"reasoning_tokens":2487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:59:12.336677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regrade the same test predictions by exact pixel comparison with ground-truth images, and also measure the OCR software's accuracy on clean ground-truth output images; if exact-pixel accuracy differs substantially from the OCR-based percentages, or if the OCR reader misreads clean ground-truth digits, then the reported results are artifacts of the recognition step rather than of logic learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on mining logical patterns from visual data, the immediate precursor to the LiLi task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Existing abductive learning framework that combines perception and reasoning, which the paper contrasts with its fully data-driven approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ResNet architectures used as LPN baselines on all six datasets."},{"cited_title":"Graves, Long short-term memory, Neural Computation 9 (8) (1997) 1735–1780","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM module used in the CNN-LSTM baseline."},{"cited_title":"Hoshen, S","cited_arxiv_id":null,"evidence_quote":"Earlier visual arithmetic learning result that provides the MLP baseline and the task framing."},{"cited_title":"Smith, An overview of the tesseract ocr engine, in: Ninth International Conference on Document Analysis and Recognition, V ol","cited_arxiv_id":null,"evidence_quote":"OCR software used to read digits from predicted images, defining the accuracy metric for every reported percentage."}],"review_version":1}