{"id":"fb623dab-b31f-4fef-a804-d082a18284f6","arxiv_id":"1908.09183","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-parameter logistic curve relating MNIST image width in pixels to human classification error is fitted, but is presented without raw data, error bars, or validation.","lead":"The authors fit a sigmoid curve to human error rates on downsampled MNIST digits and report that error depends on image width through a two-constant formula. They propose this as a planning heuristic for choosing training-image resolution in machine learning projects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (2) rests on an unvalidated two-parameter fit: no per-resolution data, error bars, or independent validation are reported, so the claimed quantitative relationship is not reproducible.","rationale":"The paper does real empirical work: building HICEAA, collecting 10,000 MNIST labels, and identifying a plausible sigmoid shape. I am not disputing the authors' effort or integrity. The reader's REJECT verdict is justified, but I would sharpen the weakest assumption. The deepest issue is not only that pixel width cannot absorb display size and resampling method, though that is also true; it is that Eq. (2) is presented as a result without the per-resolution data, trial counts, error bars, or regression diagnostics needed to distinguish a stable quantitative law from a fortuitous fit. The two-parameter sigmoid could be correct for MNIST under the exact protocol, but nothing in the manuscript lets a reader check. Cross-validation on the released per-resolution data is the decisive test: if the curve is stable under leaving out resolutions, a CONDITIONAL accept with data release would become plausible; if not, the REJECT stands. The display-size and resampling-method invariance claims are secondary and would require separate experiments varying those conditions. My agreement with the reader is partial because the reader emphasized generalization and display size, while I see the more immediate, logically prior problem as the absence of statistical evidence for the fitted curve itself.","tokens_in":6524,"tokens_out":5220,"duration_ms":55216,"concrete_test":"Obtain the raw HICEAA per-resolution accuracy table (the data behind Figures 9 and 10), refit y = 1 / (1 + exp(-(alpha*x + c))) with per-resolution denominators, and compute 95% confidence intervals for alpha and c; then leave out each resolution in turn and compare held-out predictions to observed error. If the confidence intervals do not contain alpha = -0.95 and c = 6.5, or if held-out resolutions deviate by more than roughly 3-5 percentage points at multiple resolutions, Eq. (2) should be treated as an overfit summary rather than a derived relationship. If the raw data cannot be supplied, the equation is not independently checkable and the central claim is unverdictable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that error vs. resolution was plotted, an inverse sigmoid was 'observed', and the constants alpha=-0.95 and c=6.5 were 'arrived at' (Eq. 2). This is the central quantitative claim. The load-bearing problem is that the manuscript provides no way to verify that this two-parameter logistic is more than a curve drawn through noisy data: the referenced per-resolution aggregate tables (Figures 9 and 10) are not reproduced, and no subject count, trial count per resolution, standard errors, confidence intervals, regression diagnostics, or code are provided. A curve fitted and evaluated on the same data cannot by itself support 'deriving' a general relationship. Additionally, the protocol fixes display size at 3.25 x 3.25 inches and one resampling method (cv2 INTER_AREA, Section 3.3), so the fitted constants absorb those conditions, while Eq. (2) contains only pixel width x and therefore implicitly claims an invariance the experiment never varied. The extension to remote sensing, medical imaging, and astronomy is an extrapolation from one digit-recognition dataset. The claim would be secure only if per-resolution data are released and the fit is shown to be stable under cross-validation and on at least one independent dataset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical study in which human participants classified down-sampled MNIST digits at 28 resolutions from 1x1 to 28x28 pixels. Based on 10,000 collected labels, the authors plot human classification error versus image width and claim that the relationship is an inverse sigmoid, yielding Eq. (2): y = f(x) = 1 / (1 + exp(-(alpha*x + c))) with alpha = -0.95 and c = 6.5. The paper argues that this quantitative heuristic can be used to estimate required data resolution, predict model performance, and substitute for the variable N_f in standard machine vision resolution formulas, with potential applications in remote sensing, medical imaging, and astronomy.","tokens_in":6772,"tokens_out":5743,"duration_ms":59302,"significance":"If Eq. (2) were a validated general relationship, it would provide a practically useful heuristic for planning image resolution in data labeling and machine learning projects. The experimental idea is reasonable, and the use of a public dataset (MNIST) makes the protocol easy to describe and potentially repeatable. However, the central quantitative claim is not yet supported: the paper provides no per-resolution error data, no fit diagnostics, no uncertainty quantification, and no independent validation. The extension to other modalities is pure extrapolation from a single digit-recognition task. The authors themselves state that they plan further testing on additional datasets, which underscores the preliminary character of the result.","major_comments":[{"comment":"The central result is presented with only the fitted constants alpha = -0.95 and c = 6.5. The manuscript does not report the per-resolution error rates, the number of participants, the number of trials per resolution, standard errors, confidence intervals, or any regression diagnostics. Figures 9 and 10, which are captioned as containing the aggregate error tables, are not actually included in the manuscript. Without these data, the fit cannot be assessed, reproduced, or compared with any alternative functional form, so the quantitative claim in Eq. (2) is not verifiable from the paper.","section":"§4, Eq. (2)"},{"comment":"The inverse sigmoid functional form was chosen after inspecting the plotted data, and the two constants were fitted to the same data set that Eq. (2) then claims to describe. No cross-validation, holdout set, or independent dataset is used. As a result, Eq. (2) is a description of the training points, not a derived or predicted relationship; it has not been shown to generalize to new labelers, new digit samples, or other image content.","section":"§4 and §3.3"},{"comment":"The protocol fixes the display size at 3.25 x 3.25 inches and uses a single resampling method (cv2 INTER_AREA), but Eq. (2) contains only the pixel width x. The fitted constants therefore absorb the particular display geometry, viewing conditions, and resampling algorithm. The equation implicitly claims that human classification error is determined by pixel width alone, a premise that the experiment never varied. Without explicitly stating these boundary conditions, the constants cannot be transferred to other display sizes or resampling methods.","section":"§3.3 and §4"},{"comment":"The discussion and conclusion extend the result to remote sensing, medical imaging, scientific imaging, and astronomy, but the experiment is confined to MNIST with ten digit classes and a fixed native image scale. The paper provides no argument or evidence that the sigmoid shape or the specific constants alpha and c carry over to other content types, class counts, image statistics, or imaging modalities. This extrapolation is load-bearing for the practical claims about data-acquisition requirements and model-performance prediction.","section":"§5 and §6"}],"minor_comments":[{"comment":"The phrase \"scientific imagine\" should be \"scientific imaging\".","section":"§2.4"},{"comment":"The text states that 10,000 labels were collected but does not state how many participants took part or how labels were distributed across resolutions; adding this basic methodological detail would improve clarity.","section":"§3.3"},{"comment":"The term \"resolution\" is used in different senses: in §2.2 it refers to the smallest detectable object or ground sampling distance, while in the experiment it refers to image width in pixels. The manuscript should clarify this distinction and define \"pixels/object\" explicitly.","section":"§2.2 and §3.3"},{"comment":"The caption \"Current state of the art machine learning project process\" should be hyphenated as \"state-of-the-art\".","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a clearly written preliminary report on a potentially useful heuristic, but the central quantitative claim is not supported by the evidence presented. The absence of the per-resolution data tables (Figures 9 and 10) is a major omission that prevents any evaluation of Eq. (2). The lack of independent validation and the narrow experimental basis for the cross-domain claims would need to be addressed before the manuscript could be considered for publication. A revision that releases the data, adds fit diagnostics and cross-validation, and reports a validation on at least one additional dataset would substantially strengthen the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:1908.09183. The paper does one small useful thing: collects 10,000 human labels on downsampled MNIST and fits a logistic curve to error vs pixel width. That specific curve (alpha=-0.95, c=6.5) is new in the narrow sense—I don't know another paper with those constants. The experiment is straightforward and the write-up is readable. The NIIRS and machine-vision literature review is relevant and not padded.\n\nThe soft spot is the gap between what they did and what they claim. Equation (2) is called a derived quantitative relationship, but it is a two-parameter fit with no supporting data shown. No per-resolution error table, no participant count, no error bars, no regression diagnostics, no code. The sigmoid form is 'observed' after plotting, the constants are arrived at from the same data, and there is no independent test. That makes the central claim unreproducible as presented. It is not so much circular as unvalidated; any smooth curve through averaged data would look like that.\n\nThe generalization is also overreached. The images are always displayed at 3.25×3.25 inches, resampled with INTER_AREA, and the task is MNIST digit recognition. Those conditions are baked into alpha and c, yet Eq. (2) contains only pixel width x. The claims about remote sensing, medical imaging, and astronomy are extrapolation from one toy dataset.\n\nThe paper is honest about being a heuristic and about limitations in section 5, but that does not carry the weight. I also noticed some typos and redundancy, but that is minor.\n\nWho is this for? A reader interested in ML labeling workflows might find the experimental idea worth knowing about, but the actual curve shouldn't be used until it's validated and the data released. The right next step is a revised version with per-resolution data, variance, cross-validation, and at least one independent dataset. As is, I'd not send it to a rigorous peer review; it's a workshop poster at best. If they ship the data and show the fit holds elsewhere, I'd revisit.","headline":"A useful but overreaching empirical curve fit: the central logistic equation lacks supporting data, error bars, and any independent validation.","tokens_in":7265,"tokens_out":2278,"would_cite":false,"duration_ms":23573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human classification error on square images is a logistic function of image width in pixels, fitted from 10,000 labels on down-sampled MNIST digits.","keywords":["resolution","human classification error","logistic curve","MNIST","data labeling","image quality","machine learning workflow"],"falsifier":"Present the same down-sampled MNIST images at the same pixel widths but at a materially different physical display size, such as 1.5 by 1.5 inches, and measure human error; if error rates deviate from the fitted logistic curve at matched pixel widths, the claim that pixel width alone determines error is falsified. Re-running the experiment with a different resampling method, such as nearest-neighbor instead of area averaging, and finding that $\\alpha$ and $c$ shift would also settle the question.","tokens_in":6189,"feed_emoji":"📉","tokens_out":8724,"duration_ms":79269,"temperature":0.7,"pith_summary":"This paper tries to establish a simple quantitative rule: as digital images are made coarser, human accuracy in classifying them falls along a predictable logistic curve, so the pixel width of an image can be used to estimate the human error rate before any model is trained. The authors run a controlled labeling experiment on down-sampled MNIST digits, collect 10,000 human labels across 28 resolutions, and fit a two-parameter logistic function to the aggregate error. If the curve holds, project teams could decide in advance what resolution of data to collect or label for a target level of accuracy, and could estimate the ceiling for a trained model from human performance. The practical payoff is cost and storage savings in machine-learning pipelines, plus a pre-training feasibility check that today's image-quality scales do not provide.","feed_headline":"Human error follows one predictable curve as image width grows","feed_subtitle":"A 10,000-label experiment maps resolution to error, so teams can size data before buying it.","key_machinery":"The load-bearing object is the fitted inverse-sigmoid logistic curve, $y = 1/(1 + \\exp(-(\\alpha x + c)))$ with $\\alpha = -0.95$ and $c = 6.5$, parameterized by image width $x$ in pixels. It was obtained by repeatedly resizing MNIST digits from $1 \\times 1$ to $28 \\times 28$ with area-based decimation, fixing the displayed image at $3.25 \\times 3.25$ inches, collecting 10,000 human selections, and aggregating error by resolution. The curve converts resolution into an error estimate, which is what lets the authors invert the relationship and solve for the resolution required to hit a specific human-level performance target.","core_discovery":"The paper's central claim is that human classification error rate $y$ on square images is described by $y = f(x) = 1/(1 + \\exp(-(\\alpha x + c)))$ with $\\alpha = -0.95$ and $c = 6.5$, where $x$ is image width in pixels. Because $\\alpha$ is negative, error is near certain at very coarse resolutions and falls to near zero by $28 \\times 28$, matching the intuition that tiny thumbnails become unreadable. The authors report this as an empirically derived heuristic, not a theoretical law, and propose that it can stand in for the missing pixel-count variable in standard machine-vision resolution formulas, letting a practitioner solve for the resolution needed to reach a target error rate.","pith_inferences":["The fitted constants are entangled with the fixed $3.25 \\times 3.25$-inch display used in the experiment; changing physical display size or viewing distance should shift them, so practical use outside the test setup requires an explicit display-scale term or a re-fit at the deployment display size.","If the logistic shape is generic, the same protocol could measure how much resolution a new domain needs by collecting a few hundred labels at two or three widths and checking whether they fall on the family of curves; the fitted center would then be a task-difficulty index.","Because the paper treats human error as a proxy for Bayes error, the monotone curve implies a model trained at high resolution and evaluated on down-sampled inputs should be bounded by the human error at that lower resolution, offering a cheap sanity check before deploying at reduced resolution."],"forward_implications":["The inverse of the logistic curve gives the minimum image width needed to keep expected labeling error at or below any chosen level.","Before training, a team can estimate the ceiling on model accuracy from human error at the resolution of already-available data, turning feasibility analysis into a quick calculation.","The fitted pixels-per-object value supplies the missing $N_f$ term in the standard camera-resolution formula, so the heuristic connects directly to sensor and data-collection choices.","The curve is steepest around roughly 7 pixels of image width, so reducing resolution below that point is where labeling error rises fastest and should be avoided unless unavoidable.","Using this heuristic before data acquisition lets organizations compare predicted performance against collection and labeling costs, potentially avoiding wasted expenditure on data that cannot meet the target."],"supporting_citations":[{"why":"Establishes the premise that human-level error approximates Bayes error, which is why human performance can serve as a model ceiling.","marker":"[16]"},{"why":"Documents the resolution method for aerial image quality and its limitation of not quantifying interpretability versus resolution.","marker":"[17]"},{"why":"Represents state-of-the-art NIIRS-based interpretability measures that nonetheless lack a quantitative resolution-error relationship.","marker":"[10, 11, 18]"},{"why":"Supplies the standard machine-vision camera resolution formula whose missing pixel-count term the paper proposes to fill.","marker":"[8]"},{"why":"Previous study on how many pixels are needed to see objects; the paper notes it did not control image scale, motivating the new controlled experiment.","marker":"[2]"},{"why":"Basis for the claim that humans segment images consistently, supporting the use of non-representative test subjects.","marker":"[12]"}],"fun_headline_variants":["Why tiny images stump labelers: the math behind resolution","One formula links image width to human misclassification rate","Blur's cost on human vision: one equation explains it all","From unreadable to flawless: the resolution-error curve","A quantifiable link between pixel size and human accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that human classification error is fully determined by image width in pixels, independent of display size, viewing distance, resampling method, and dataset content, so the fitted logistic constants generalize beyond the test setup.","fun_headline_variants_meta":{"raw":{"variants":["Why tiny images stump labelers: the math behind resolution","One formula links image width to human misclassification rate","Blur's cost on human vision: one equation explains it all","From unreadable to flawless: the resolution-error curve","A quantifiable link between pixel size and human accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1526,"prompt_tokens":853,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":592}},"tokens_in":469,"tokens_out":673,"duration_ms":7744,"temperature":1.0,"reasoning_tokens":592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:18:08.533213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present the same down-sampled MNIST images at the same pixel widths but at a materially different physical display size, such as 1.5 by 1.5 inches, and measure human error; if error rates deviate from the fitted logistic curve at matched pixel widths, the claim that pixel width alone determines error is falsified. Re-running the experiment with a different resampling method, such as nearest-neighbor instead of area averaging, and finding that $\\alpha$ and $c$ shift would also settle the question.","supporting_citations":[{"cited_title":"Deep Learning: Structuring Machine Learning Projects","cited_arxiv_id":null,"evidence_quote":"Establishes the premise that human-level error approximates Bayes error, which is why human performance can serve as a model ceiling."},{"cited_title":"Comparison of two common aerial reconnaissance image quality measures","cited_arxiv_id":null,"evidence_quote":"Documents the resolution method for aerial image quality and its limitation of not quantifying interpretability versus resolution."},{"cited_title":"Handbook of Machine and Computer Vision: The Guide for Developers and Users","cited_arxiv_id":null,"evidence_quote":"Supplies the standard machine-vision camera resolution formula whose missing pixel-count term the paper proposes to fill."},{"cited_title":"How Many Pixels Do We Need to See Things?","cited_arxiv_id":null,"evidence_quote":"Previous study on how many pixels are needed to see objects; the paper notes it did not control image scale, motivating the new controlled experiment."},{"cited_title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","cited_arxiv_id":null,"evidence_quote":"Basis for the claim that humans segment images consistently, supporting the use of non-representative test subjects."}],"review_version":1}