{"id":"0f29212d-2a2c-4336-a255-8b074e46e394","arxiv_id":"2505.10862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4.1 reads standard analog clocks well but fails badly on visually altered clocks, indicating memorization of clock images rather than a general ability to tell time.","lead":"Multimodal AI models that can answer complex visual questions still misread analog clocks when the clock face or hands are altered. This paper tests GPT-4.1 on normal, deformed, and arrow-handed clocks and finds accuracy collapses, suggesting the model memorized training patterns rather than learned to tell time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing human baseline leaves the 'memorization, not learning' conclusion unsupported; the MAE gap could reflect stimulus difficulty rather than lack of abstraction.","rationale":"The paper is a compact empirical study with public data; the main MAE pattern is large and directionally plausible. I agree with the reader that the weakest point is the unverified human-readability premise. It is load-bearing because the headline conclusion is explicitly about what the model has not learned, and that contrast class is human abstraction. A 3,700s MAE on arrow-handed clocks is only evidence of memorization rather than rule-learning if the arrow-handed clocks are trivially readable by a competent visual reasoner. The paper's own single-hand experiment is a useful partial control, but it neither uses full clocks nor provides human data. I would keep the reader's CONDITIONAL verdict: the empirical phenomenon is likely robust, but the interpretation should be conditional on a human baseline. Secondary issues (excluding six samples, underspecified fine-tuning splits) affect specific numbers but are not as central.","tokens_in":5213,"tokens_out":4184,"duration_ms":46494,"concrete_test":"Run a human baseline on the released dataset: show the same 150 normal, 150 deformed, and 150 arrow-handed clock images used for the MAE values to a group of, say, 20 participants, collect their reported times, and score with the same MAE and hand-confusion protocol used for GPT-4.1. If human MAE on deformed and arrow-handed clocks is near zero and hand-function confusion is rare, the memorization interpretation is supported; if humans also show elevated error or hand confusion on these variants, the performance gap is confounded by stimulus difficulty and the paper's claim cannot be sustained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference in Section 2—that GPT-4.1's MAE jump from 232.48s (normal) to 1380.69s (deformed) and 3726.93s (arrow hands) shows the model 'has not learned to tell the time but rather memorized patterns'—depends on the unmeasured premise that the modified clocks are as easy for humans as normal clocks. The text asserts this ('A person would be able to tell the time in these clocks with no difficulty') but provides no human baseline or perceptual sanity check. Without that control, the large MAE gaps are equally consistent with the modified images being genuinely more ambiguous (e.g., arrowheads obscuring hand direction or deformed dials altering tick alignment), in which case GPT-4.1's behavior reflects stimulus difficulty, not memorization versus abstraction. The single-hand probe in Section 4 partially addresses directional perception (6.5° vs 8.1° MAE), but it does not test the full-clock images used for the headline claim, nor does it establish human-level readability. Consequently the strongest claim is underdetermined by the reported data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether GPT-4.1 has learned a general rule for reading analog clocks or has merely memorized patterns from training images. The authors construct a synthetic clock dataset, measure GPT-4.1's mean absolute error (MAE) on normal clocks (232.48 seconds), deformed \"Dalí-style\" dials (1380.69 seconds), and clocks with thinner arrow-headed hands (3726.93 seconds), and then fine-tune the model on 300 samples per clock type. After fine-tuning, MAEs improve to 33.82, 784.41, and 2893.57 seconds, respectively. The authors interpret the large performance drop on visually altered clocks as evidence that the model has not learned to tell time but has memorized patterns, and they diagnose two failure modes: confusion between hand functions and reduced directional sensitivity. A single-hand probing experiment and a comparison of confused versus non-confused samples are used to support this interpretation. The manuscript is a short empirical case study with an open dataset and code repository.","tokens_in":5368,"tokens_out":3933,"duration_ms":40830,"significance":"If the central claim survives additional controls, the paper would be a useful and timely case study of MLLM brittleness in a simple spatial reasoning task, showing that apparent competence on synthetic in-distribution images does not imply abstraction and that fine-tuning provides limited transfer to perceptual variants. The strengths are the clear task design, the release of the dataset and code, the before/after fine-tuning comparison, and the decomposition of errors into hand-function confusion versus directional error. However, the paper's headline conclusion is currently underdetermined: the inference from large MAE gaps to 'memorization, not learning' requires a human baseline or perceptual sanity check, and the reported numbers lack uncertainty quantification. The significance of the result therefore depends on the outcome of those controls, rather than being established by the present data.","major_comments":[{"comment":"The central conclusion that the MLLM 'has not learned to tell the time but rather memorized patterns' relies on the premise stated in Section 2 that 'A person would be able to tell the time in these clocks with no difficulty.' No human baseline or perceptual sanity check is reported for the deformed and arrow-handed stimuli. Without such a control, the MAE increases (232.48s to 1380.69s and 3726.93s) are equally consistent with the modified images being more ambiguous even for humans, for example because arrowheads obscure hand direction or deformed dials alter tick alignment. This is not a circularity issue, but the memorization-versus-abstraction interpretation is underdetermined. A concrete fix is to measure human MAE on the same image sets, or at least to run a perceptual task confirming that the altered images are readable by humans; if human performance also degrades on these stimuli, the central interpretation must be revised.","section":"Section 2"},{"comment":"All headline MAE numbers come from a single evaluation set of 150 random times, with no confidence intervals, repeated trials, or per-sample variance reported. Given the large spread within the data (Table 1 shows MAEs of 10088.9s for confused arrow-hand samples versus 882.2s for non-confused samples), the claimed 16x degradation between normal and arrow-handed clocks needs uncertainty quantification. Bootstrap confidence intervals over the 150 samples, or repeated evaluation with different random seeds, are necessary to establish that the gaps are not driven by a small number of outliers.","section":"Section 2"},{"comment":"The fine-tuning protocol is under-specified. The text states that the model was fine-tuned with 'another set of 300 random samples from each of the datasets' and then evaluated on 'the 150 examples,' but it does not state whether the 150 evaluation times are disjoint from the 300 training times, which base model checkpoint was used for fine-tuning, or what hyperparameters (number of steps, learning rate, LoRA versus full fine-tuning) were used. If the evaluation times overlap with the fine-tuning examples, the post-fine-tuning improvements (33.82s, 784.41s, 2893.57s) could reflect memorization rather than generalization. Please report the exact split and fine-tuning configuration.","section":"Section 2"},{"comment":"The exclusion of six samples because 'the reason of the error could not be clearly determined' is a post hoc filtering step with no stated criterion or inter-annotator agreement. This can bias the comparison between the 'confused' and 'not confused' groups, and the resulting group sizes (45, 99, and 6 excluded) are small. In addition, the single-hand probe in Section 4 tests only the hour hand on 60 images per hand type and reports MAEs of 6.5 degrees versus 8.1 degrees without confidence intervals; this is insufficient to rule out a directional-perception contribution to the full-clock failure. Please provide the classification protocol, the reasons for exclusion, and uncertainty estimates for the angular error comparisons.","section":"Section 4 and Table 1"}],"minor_comments":[{"comment":"The text 'As shown in Figure 2, this issue accounts for all the errors in the normal clock dataset' appears to reference the wrong figure: Figure 2 is a scatter plot of predicted versus true time, whereas the error-type analysis is presented in Table 1 and Figure 4.","section":"Section 3"},{"comment":"The second table caption repeats the first table's caption ('Table 2: Analysis of samples with and without clock hand function confusion'); the second table should be captioned to describe the angular errors per hand role.","section":"Section 4"},{"comment":"There is a typo in 'thar are trivial for humans'; it should read 'that are trivial for humans.'","section":"Section 5"},{"comment":"Figure 5 is described as an overview of performance but the caption does not specify what the axes, bars, and any error markers represent; please add a self-contained caption.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the missing human baseline does land: the paper's central interpretation is currently underdetermined. The study is otherwise a well-scoped empirical case study with useful open artifacts, so I see this as a fixable issue rather than a fatal one. No concerns about citation practice or scope beyond the need to treat the general 'memorization' claim more cautiously."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper has a real, clean observation: GPT-4.1 reads synthetic analog clocks well, but MAE goes from 232s on normal clocks to 1381s on deformed dials and 3727s on arrowhead hands, and fine-tuning only partially recovers. Second, the leap from that observation to \"the MLLM has not learned to tell the time but rather memorized patterns\" is under-supported: the authors assert that humans would have no difficulty with the modified clocks, but they provide no human baseline or perceptual difficulty measure, and the reported numbers come from single runs with no error bars.\n\nWhat's genuinely new: the specific perturbations (deformed dials, thin arrowhead hands) and the decomposition of failures into direction errors versus hand-role confusion. The single-hand probe (60 images per hand type, 6.5° vs 8.1° MAE) is a reasonable sanity check showing that direction perception alone doesn't explain the full-clock collapse, though it doesn't test the exact full-clock images from the main experiment. The public dataset and code links are a real plus. Fine-tuning results (good gains on normal clocks, smaller gains on modified hands) are a useful data point for anyone working on multimodal fine-tuning.\n\nWhere it bends: the \"memorization\" conclusion is the paper's selling point, and it needs a human baseline. The authors could also add repeated trials or per-sample variance. Six excluded samples in the confusion analysis are post hoc; the authors should say why they were excluded before seeing the errors, or show sensitivity to the exclusion. The fine-tuning section says \"the model\" without clarifying whether they fine-tuned GPT-4o or GPT-4.1—the text mentions GPT-4o earlier and GPT-4.1 later, and Figure 3 refers to original and fine-tuned versions of GPT-4.1. That needs to be explicit. Also, the 150 evaluation times should be shown to be disjoint from the 300 fine-tuning samples.\n\nNone of this kills the paper. The large MAE gaps are robust enough to support the weaker claim that analog-clock reading in MLLMs is brittle under local visual changes, and that fine-tuning doesn't efficiently generalize. The stronger \"memorization, not learning\" claim should be flagged as an interpretation, not a demonstrated result.\n\nWho this is for: researchers evaluating multimodal robustness and fine-tuning, and anyone building benchmarks for spatial reasoning in MLLMs. It's a short case study, but a useful one. I'd accept it for peer review with expectations of heavy revision, and I'd encourage the authors to add a human baseline and temper the abstract.","headline":"A useful perturbation study showing GPT-4.1 is brittle at analog-clock reading, but the 'memorization, not learning' conclusion needs a human baseline.","tokens_in":5919,"tokens_out":3680,"would_cite":true,"duration_ms":31634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a model that reads ordinary analog clocks well is only matching training patterns, not learning the time-telling rule.","keywords":["multimodal large language models","analog clock reading","memorization versus generalization","fine-tuning","spatial reasoning","synthetic clock dataset","hand role confusion","directional perception"],"falsifier":"Have human readers tell the time on the same 150 synthetic clock images for all three variants. If human mean absolute error on the deformed and arrow-handed clocks stays near zero while the model's is in the hundreds or thousands of seconds, the memorization conclusion is supported; if humans also make large errors on those variants, the conclusion collapses because the images are not trivially readable. A second check would be to fine-tune on clocks with hand shape and thickness randomized independently of hand role and then test on a never-seen hand style; a model that has learned the abstract rule should transfer, while a pattern-memorizer should not.","tokens_in":4974,"feed_emoji":"🕰️","tokens_out":8180,"duration_ms":80106,"temperature":0.7,"pith_summary":"The paper asks whether a state-of-the-art multimodal large language model that can read ordinary analog clocks has genuinely learned the skill, or has instead memorized surface patterns from its training images. It reports that, when the same clock faces are deformed or their hands are made thin and arrow-headed, the model's mean absolute error jumps from about 232 seconds to about 1381 seconds and 3727 seconds, respectively. Fine-tuning on extra synthetic samples lowers the normal-clock error to about 34 seconds but leaves the deformed and arrow-handed errors at roughly 784 and 2894 seconds. The authors read this as evidence that the model has not abstracted the time-telling rule, and they trace the failures to two mechanisms: poor directional perception and confusion between hour and minute hands caused by reliance on hand appearance. The wider claim is that current MLLMs can be brittle on simple spatial reasoning, and that fine-tuning with more examples is an inefficient patch for that brittleness.","feed_headline":"MLLMs memorize analog clocks instead of learning to tell time","feed_subtitle":"Changing dial shape and hand style raises mean error from 4 minutes to over an hour; fine-tuning only trims the gap.","key_machinery":"The working instrument is a synthetic analog-clock dataset in which the same time is rendered three ways: a normal dial, a distorted dial, and a dial with thin arrow-headed hands. By keeping the clock-reading task fixed and changing only visual surface features, the dataset separates generalization from memorization and lets the authors isolate two error channels: misreading a hand's direction, measured as angular error per hand, and confusing which hand is the hour, minute, or second hand. A second probe uses isolated hour hands pointing at each of the 60 tick marks, with normal versus modified hands, to test whether hand shape impairs direction perception on its own. Together these instruments show that hand-role confusion, not pure direction error, is the dominant cause of the catastrophic drop on arrow-handed clocks, and that even when that confusion is removed the model still fails to transfer the benefit of fine-tuning.","core_discovery":"The central discovery is that the MLLM's apparent competence on normal analog clocks does not survive small visual transformations that leave the underlying time information unchanged. In the paper's measurements, GPT-4.1 attains a mean absolute error of 232.48 seconds on the 150 normal clock images, 1380.69 seconds on distorted-dial clocks, and 3726.93 seconds on clocks with thinner, arrow-headed hands. After fine-tuning with additional synthetic samples the corresponding errors are 33.82, 784.41, and 2893.57 seconds, so the intervention helps most where the model was already good and least where it was struggling. Error decomposition shows that direction errors dominate on normal and deformed clocks, while on arrow-handed clocks the model frequently swaps the roles of hour and minute hands. Even after excluding those swaps, fine-tuning reduces error by only 32.5 percent, leading the authors to conclude that the model has memorized training patterns rather than learned to tell time, and that its failures stem from low directional sensitivity plus overfitting to hand appearance.","pith_inferences":["If the paper's interpretation is right, a controlled human test on the exact same 150 clock images would separate perceptually ambiguous stimuli from genuine memorization; near-human accuracy on the deformed and arrow-handed clocks would support the memorization claim, while large human errors would undermine it.","The same controlled-deformation methodology could be applied to other seemingly trivial visual tasks, such as reading analog pressure gauges or interpreting protractors, to test whether MLLMs are brittle to style changes whenever spatial-attribute integration is required.","A training intervention that randomizes hand shape, thickness, length, and color independently of hand function during fine-tuning would reveal whether the model can learn a function-independent rule; the paper's findings predict such randomization is necessary for generalization.","A sharper memorization test would use clock styles entirely absent from training data, such as clocks with no tick marks or with reversed handedness; a true rule-learner should transfer, while a pattern-memorizer should fail."],"forward_implications":["Fine-tuning on more clock images will not by itself make a model robust to other clock styles; the improvement on normal clocks is about 85 percent error reduction, versus about 43 percent on deformed clocks and about 22 percent on arrow-handed clocks.","A model can identify a hand's direction correctly in isolation yet fail on a complete clock, so fixing spatial perception alone is insufficient because functional role assignment is a separate failure point.","Small local changes to a familiar input can cause large accuracy drops even when the underlying task is unchanged, implying that MLLM performance on familiar-looking image types may overstate their competence on visually shifted inputs.","The fine-tuning bottleneck worsens when multiple sources of interference combine, so collecting larger datasets for every visual variant is not a scalable route to general multimodal reasoning."],"supporting_citations":[{"why":"It supplies the prior result that multimodal LLMs struggle with clock and calendar reading, which this paper extends with controlled clock-image variants.","marker":"[5]"},{"why":"It provides the memorization-versus-rule-learning probing framework the paper uses to interpret the error gaps across clock variants.","marker":"[8]"},{"why":"It supports the analogy that adding training examples can fix rule-like failures such as arithmetic in language models, motivating the fine-tuning experiment.","marker":"[6]"},{"why":"It grounds the comparison showing small models can learn arithmetic from explicitly added examples, which is why the paper expects fine-tuning to work if data scarcity were the only obstacle.","marker":"[7]"}],"fun_headline_variants":["MLLMs can't read clocks they haven't seen before","Even fine-tuned MLLMs can't tell time on new clocks","MLLM clock skills vanish with different hands","Memorized not learned: MLLMs fail on novel clock faces","Fine-tuning fails to teach MLLMs generalization to new clocks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The interpretation that the model is memorizing rather than learning depends on the unmeasured premise that a person can read the deformed and arrow-handed clocks without difficulty; if those images are perceptually ambiguous even for humans, the error gaps could just reflect stimulus difficulty rather than a failure to abstract the time-telling rule.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs can't read clocks they haven't seen before","Even fine-tuned MLLMs can't tell time on new clocks","MLLM clock skills vanish with different hands","Memorized not learned: MLLMs fail on novel clock faces","Fine-tuning fails to teach MLLMs generalization to new clocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3103,"prompt_tokens":889,"completion_tokens":2214,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2127}},"tokens_in":505,"tokens_out":2214,"duration_ms":16629,"temperature":1.0,"reasoning_tokens":2127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:01:48.442613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human readers tell the time on the same 150 synthetic clock images for all three variants. If human mean absolute error on the deformed and arrow-handed clocks stays near zero while the model's is in the hundreds or thousands of seconds, the memorization conclusion is supported; if humans also make large errors on those variants, the conclusion collapses because the images are not trivially readable. A second check would be to fine-tune on clocks with hand shape and thickness randomized independently of hand role and then test on a never-seen hand style; a model that has learned the abstract rule should transfer, while a pattern-memorizer should not.","supporting_citations":[{"cited_title":"Lee, Kangwook Lee, and Dimitris Papailiopoulos","cited_arxiv_id":null,"evidence_quote":"It grounds the comparison showing small models can learn arithmetic from explicitly added examples, which is why the paper expects fine-tuning to work if data scarcity were the only obstacle."}],"review_version":1}