{"id":"11d6bbc1-da28-4316-90a1-218ad5b4a32a","arxiv_id":"1909.04147","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A three-parameter linear model using change magnitude, salience imbalance, and user experience predicts mode detection times for color change blindness in cartoon images with correlation 0.62.","lead":"The authors built a benchmark of 100 cartoon image pairs with color changes and measured how long 60 people took to notice each change. They then built a simple three-feature model, including how many images a person had already seen, that predicts average detection difficulty with moderate accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central results may hinge on post hoc choice of the mode target and the 36 s density-minimum threshold; an a priori target or threshold check is needed.","rationale":"I read the paper in good faith and find it honest and useful: it provides a new benchmark, a simple three-parameter model, and it does not overclaim individual-level prediction, reporting PLCC 0.29 for individual detection times. The load-bearing weakness is exactly what the reader identified: the mode is adopted as the target after observing that it yields two clean clusters, and the classification threshold is the minimum density of that same mode distribution. Because the regression and classification results are only evaluated on this post hoc target, the headline numbers may overstate how well the model would perform for a pre-specified, standard summary such as the mean or median. This is a correctness risk rather than a proven failure: the model could still rank difficulty well under an a priori target, so the appropriate disposition is conditional acceptance pending the concrete check. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":10666,"tokens_out":4973,"duration_ms":60253,"concrete_test":"Re-run the regression of Section IV-A and the classification of Section IV-B using the mean (or median) detection time as the target instead of the mode, and using a pre-registered threshold (e.g., 30 s or the median of all individual DTs) instead of the post hoc density-minimum threshold, keeping the same features, splits, and cross-validation procedure. If the mode-target PLCC/SROCC drop from 0.62/0.63 to the ~0.3 level of individual-DT prediction, or if the 85% accuracy falls substantially, then the central claim depends on the post hoc target choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest results—PLCC 0.62 / SROCC 0.63 for regression and 85% classification accuracy—are reported for a target that was chosen after inspecting the data. In Section III-A2, the authors observe that detection-time distributions are non-normal and often multimodal, and then select the dominant mode \\hat{T}_i as the regression target because the mean does not produce clean clusters. The two difficulty classes C1 and C2 are then defined by Tcrit = 36 s, placed at the minimum density of the mode distribution, effectively using the same post hoc observation to create the labels that classification later predicts (Section IV-B uses 35 s, a minor inconsistency). The model's correlation with individual detection times is only 0.29 PLCC, so the practical claim rests entirely on the mode-based target. With only 21–41 observations per image pair, the mode estimate itself may be unstable, and a threshold chosen at the density minimum could make labels appear more separable than they are for any pre-specified target. This is not an internal error in the regression or cross-validation; it is a concern about whether the reported predictive power would survive an a priori choice of the summary statistic and difficulty threshold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a three-feature linear regression model for predicting colour change blindness in cartoon image pairs viewed under a flicker paradigm. The features are change magnitude (mean perceived colour difference over the changed pixels), salience imbalance (Hamming distance between DCT sign patterns of the two images, averaged over colour channels), and user experience (squared average of the stimulus rank in each observer's random sequence). The target statistic is the dominant mode of the per-pair detection-time distribution, selected after observing that detection times are non-normal and often multimodal. On a dataset of 100 image pairs and 60 valid observers, the model is reported to predict mode detection times with PLCC 0.62 and SROCC 0.63 under 100 random 70/30 splits, outperforming individual features, the Ma et al. model, and other regressors. The paper also reports binary classification of stimuli into easy/hard classes with 85% accuracy using quadratic discriminant analysis with ten-fold cross-validation, where the difficulty threshold is placed at the density minimum of the mode distribution.","tokens_in":10960,"tokens_out":2293,"duration_ms":25004,"significance":"The paper makes two concrete contributions: a new publicly available change-blindness benchmark with controlled collection conditions and observer ranks, and a simple, fully automatic predictive model with only three parameters. The strengths are the clean experimental setup, the explicit cross-validation protocol with 100 random splits, the fair comparison with several baselines, and the placement of the dataset in the public domain. If the mode-based target and the threshold-based difficulty classes were justified a priori, the model would be a practically useful tool for ranking and classifying the difficulty of spot-the-difference stimuli. However, the central quantitative claims (correlations of approximately 0.62 and classification accuracy of 85%) are obtained for a target statistic and a threshold that were chosen after inspecting the same data used for evaluation, and the mode estimate itself is noisy given only 21-41 observations per pair. This post hoc selection is the main threat to the validity of the reported performance and should be addressed before the results can be taken at face value.","major_comments":[{"comment":"The choice of the dominant mode as the regression target is made post hoc. Section III-A2 states that the mean detection time does not produce clearly separated clusters, and therefore the dominant mode is selected as the target. This means the fitted regression target in Eq. (5) is itself a statistic chosen from the observed data. With only 21-41 observations per image pair, the sample mode is unstable, and a target selected to produce clean clusters can inflate the reported correlations (Table II, PLCC 0.62 / SROCC 0.63). Please provide an a priori definition of the target statistic, or validate the mode-based analysis on an independent hold-out set with the target fixed before any model fitting.","section":"III-A2"},{"comment":"The classification threshold is inconsistent and data-derived. Section III-A2 defines Tcrit = 36 s as the global density minimum of the mode distribution, but Section IV-B reports classification using a 35 s threshold, and Figure 3's caption mentions 'around 35s'. Moreover, the threshold is derived from the same data used to evaluate the 85% classification accuracy. Because the classes C1 and C2 are defined by this threshold, the accuracy partly reflects a fit to the data. Please report classification results with a pre-specified threshold (e.g., 36 s fixed before analysis, or a threshold chosen on the training folds only, with the test folds evaluated using that threshold) and clarify the discrepancy between 35 s and 36 s.","section":"III-A2 / IV-B"},{"comment":"The paper's practical claim rests on the mode-based target, yet the model's correlation with individual detection times is only PLCC 0.29 and SROCC 0.29. The abstract and introduction frame the model as 'predict[ing] colour change blindness' and 'predict[ing] detection times', which a reader would naturally interpret as individual-trial detection times. The gap between individual-level and mode-level performance should be discussed explicitly, and the claims should be qualified as applying to a summary statistic of each pair rather than to individual observers.","section":"Table II"}],"minor_comments":[{"comment":"The Kolmogorov-Smirnov test is described as being applied 'at the 5% confidence level'; the correct terminology is 'significance level'.","section":"III-A2"},{"comment":"The notation `#D` in Eq. (1) is used without a prior definition; please define it as the number of changed pixels.","section":"Eq. (1)"},{"comment":"The caption of Figure 3 says 'around 35s', while the text in Section III-A2 gives 36 s; please make the numbers consistent.","section":"Figure 3"},{"comment":"The sentence 'we trained the three parameters of Eqn. (2) in [31]' is unclear because Eq. (2) in the cited paper [31] is not reproduced here; please briefly describe the Ma et al. model or its parametrisation.","section":"Section IV-A"},{"comment":"The definition of fUE in Eq. (4) as the squared average rank deserves a one-sentence justification; as written, the squaring seems arbitrary and is not motivated in the text.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is valuable and the experimental methodology is generally sound, but the post hoc selection of the regression target and classification threshold is a substantive threat to the central claims. The authors should be asked to either fix the target and threshold a priori or provide a validation procedure that does not use the same data to define the target and evaluate the model. If they can do so, the paper would likely be suitable for publication; in its current form, the reported performance figures are not fully persuasive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful paper and you should look at it before deciding about the field. The original contribution is a new public benchmark for change blindness: 100 cartoon image pairs, 3418 valid detection times, and per-observer sequence positions. That scene-rank information is genuinely new, and it lets the authors include short-term experience as a predictor. The model itself is pleasingly simple—linear regression on change magnitude, salience imbalance, and user experience, with only three parameters—and it beats both the individual features and the Ma et al. model on their data, with cross-validated PLCC 0.62 and SROCC 0.63 for mode detection times. They are also candid that per-participant prediction is weak (PLCC 0.29) and RMSE is 14.8s. That honesty earns credit.\n\nThe main weakness sits in the target. The authors chose the dominant mode as the summary statistic after seeing that the mean does not produce clean clusters, and they set the easy/hard threshold at the density minimum of those modes. Both the regression target and the classification labels are thus derived from the same data that later gets cross-validated. The CV procedure is standard, but it cannot undo the data-driven choice of the target. With 21–41 observations per pair, the mode is not a rock-stable statistic, and moving the threshold a few seconds could change the 85% accuracy. I would want a sensitivity analysis using mean, median, or trimmed mean, and a sweep of thresholds. If the model still holds at, say, 30s and 40s, I would trust it much more.\n\nSmaller issues: the theory section uses a 36s threshold while the classification section uses 35s, a minor inconsistency that should be fixed; the 'fully automatic' claim is overstated because the change-magnitude feature requires knowing the changed pixels; and the public benchmark has no link in this arXiv version. The cartoon-only scope is fine if presented as such.\n\nBottom line: the paper deserves a serious referee. The core idea—simple prediction using scene rank—is interesting and the dataset is a contribution. The post hoc target selection is a real concern but an addressable one. I'd ask for the sensitivity analysis and the dataset link, then likely cite the benchmark.","headline":"Useful benchmark and honest simple model, but the headline correlations rest on a post hoc mode target and an empirically chosen threshold.","tokens_in":11441,"tokens_out":4902,"would_cite":true,"duration_ms":49866,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three features predict how long a colour change goes unnoticed.","keywords":["change blindness","detection time prediction","salience imbalance","user experience","linear regression","cartoon images","visual attention","colour difference"],"falsifier":"Collect detection times for a fresh, independent set of cartoon pairs using the same flicker protocol and check whether per-pair distributions still have a single dominant mode and whether those modes still form two clusters separated near 36 seconds. If the bimodal structure does not reappear, the regression target and the 85% classification accuracy are tied to this dataset; a cheaper check is to refit the three features against mean detection times and see whether the correlation advantage over individual features survives.","tokens_in":10489,"feed_emoji":"👀","tokens_out":6833,"duration_ms":59899,"temperature":0.7,"pith_summary":"The paper aims to establish that change blindness in a 'spot the difference' task can be predicted by a deliberately simple model: linear regression on just three inputs. Those inputs are the average perceptual size of the colour change, the imbalance in bottom-up salience between the two images, and the observer's short-term experience, measured by how many previous trials they have completed. On a new benchmark of 100 cartoon pairs seen by 60 observers, the model predicts each pair's dominant detection time with a Pearson correlation of 0.62 and Spearman 0.63, and it separates pairs into easy and hard classes with 85% cross-validated accuracy, outperforming any single feature and a prior change-blindness model. The practical stake is that an automatic, parameter-light difficulty score for colour changes could support image compression, visual quality assessment, user interfaces, and puzzle design.","feed_headline":"Three features predict how long a colour change goes unnoticed","feed_subtitle":"Two image features plus viewer experience rank spot-the-difference pairs by difficulty with 85% accuracy.","key_machinery":"The carrying object is the three-term linear regressor $\\tilde{T}_i = b_1 f_{CM} + b_2 f_{SI} + b_3 f_{UE}$, with only the three coefficients fitted. $f_{CM}$ is the mean LAB2000HL Euclidean colour distance over the changed pixels only; $f_{SI}$ is the average, across the three colour channels, of the Hamming distance between the signs of DCT coefficients of the original and changed images, a measure of salience imbalance; $f_{UE}$ is the squared average over observers of the pair's rank in each observer's random trial sequence, representing short-term learning. The regression target is the dominant mode $\\hat{T}_i$ of each pair's detection times, chosen because the modes split into two clusters at 36 seconds. The same feature triple, fed to quadratic discriminant analysis, is what yields the 85% easy/hard classification.","core_discovery":"The central claim is that the difficulty of noticing a single colour change is not best summarised by the mean detection time, but by the dominant mode of the detection-time distribution, and that this mode is predictable from two image-computable features plus experience. After collecting 3418 valid detection times, the paper finds every pair's distribution is non-normal with one dominant peak, and the modes form two natural clusters separated by a density minimum at about 36 seconds. Regression on change magnitude, salience imbalance, and experience reaches PLCC 0.62 and SROCC 0.63 against mode detection times, significantly better than each feature alone and than the existing model [31], which scores 0.39. The same three features, used in quadratic discriminant analysis, classify pairs into the two mode-based clusters with 85% accuracy; a four-node decision tree achieves 78% and reveals that the hardest pairs combine low experience with large colour changes, or high experience with a large change and weak salience imbalance.","pith_inferences":["If the salience-imbalance feature works because sign flips of DCT coefficients flag local spectral change, a learned salience difference on natural photographs might substitute for it, extending the same three-term model beyond cartoons.","The two mode clusters hint at two observer subpopulations — one fast and one slow — so modelling the clusters separately, rather than pooling everyone, could raise the ceiling above the reported correlations.","Because the experience feature enters as a squared average rank, the model suggests that detection-time noise grows for pairs shown out of order relative to the group average, which an experiment designer could exploit by randomising more aggressively when estimating modes."],"forward_implications":["A fully automatic pipeline can rank any collection of single-colour-change image pairs from easiest to hardest without eye tracking, manual annotation, or per-image parameter tuning.","Because the experience feature is simply trial count, adaptive experiments or games could schedule stimulus difficulty around the observer's accumulated practice.","Linear pooling of the three features is not significantly worse than support vector regression, a neural network, or a decision tree, so the operational model stays at three fitted parameters.","The released benchmark includes the per-observer stimulus order, enabling other groups to model experience and practice effects that earlier public change-blindness data lacked."],"supporting_citations":[{"why":"Defines change blindness and the flicker paradigm used to collect detection times.","marker":"[1]"},{"why":"Supplies the DCT sign Hamming distance method used as the salience imbalance feature fSI.","marker":"[33]"},{"why":"The change-blindness image model the proposed model is compared against and must beat on the new dataset.","marker":"[31]"},{"why":"Defines the LAB2000HL perceptually uniform colour space in which change magnitude is measured.","marker":"[47]"},{"why":"Prior change-blindness database that lacks per-observer stimulus order, motivating the collection of scene-rank data for the experience feature.","marker":"[42]"}],"fun_headline_variants":["Mode, not mean, predicts change blindness with 85% accuracy","Three features rank spot-the-difference difficulty precisely","Two image cues + experience: change blindness predictor","Outperform old model: 3-parameter change blindness forecast","Salience, magnitude, experience: key to change blindness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dominant mode of each pair's detection times, a statistic selected after seeing the data, is the correct measure of that pair's difficulty, and that the 36-second gap between two mode clusters is a stable property of change blindness rather than a product of this particular dataset and threshold choice.","fun_headline_variants_meta":{"raw":{"variants":["Mode, not mean, predicts change blindness with 85% accuracy","Three features rank spot-the-difference difficulty precisely","Two image cues + experience: change blindness predictor","Outperform old model: 3-parameter change blindness forecast","Salience, magnitude, experience: key to change blindness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000978,"raw_usage":{"total_tokens":4095,"prompt_tokens":827,"completion_tokens":3268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":3188}},"tokens_in":443,"tokens_out":3268,"duration_ms":25065,"temperature":1.0,"reasoning_tokens":3188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:11:41.024451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect detection times for a fresh, independent set of cartoon pairs using the same flicker protocol and check whether per-pair distributions still have a single dominant mode and whether those modes still form two clusters separated near 36 seconds. If the bimodal structure does not reappear, the regression target and the 85% classification accuracy are tied to this dataset; a cheaper check is to refit the three features against mean detection times and see whether the correlation advantage over individual features survives.","supporting_citations":[{"cited_title":"Change blindness,","cited_arxiv_id":null,"evidence_quote":"Defines change blindness and the flicker paradigm used to collect detection times."},{"cited_title":"Image signature: Highlighting sparse salient regions,","cited_arxiv_id":null,"evidence_quote":"Supplies the DCT sign Hamming distance method used as the salience imbalance feature fSI."},{"cited_title":"Change blindness images,","cited_arxiv_id":null,"evidence_quote":"The change-blindness image model the proposed model is compared against and must beat on the new dataset."},{"cited_title":"Toward a unified color space for perception-based image processing,","cited_arxiv_id":null,"evidence_quote":"Defines the LAB2000HL perceptually uniform colour space in which change magnitude is measured."},{"cited_title":"Cb database: A change blindness database for objects in natural indoor scenes,","cited_arxiv_id":null,"evidence_quote":"Prior change-blindness database that lacks per-observer stimulus order, motivating the collection of scene-rank data for the experience feature."}],"review_version":1}