{"id":"46f9ba21-57d6-4c4a-8378-08baf6b122a5","arxiv_id":"2412.05553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A psychophysical loss that sets a Gaussian center penalty using human accuracy per distance and occlusion modestly improves aerial person detection at 70 meters in NOMAD.","lead":"The authors collected human search data on thousands of aerial images from the NOMAD rescue dataset and used measured human accuracy to reshape a RetinaNet detector's bounding-box loss. On held-out NOMAD images, the adapted loss improved detection at 70 meters under occlusion without hurting close-range performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-derived σ(d,v) in Eq. (2) is computed from responses on images of the same test actors later used for evaluation; the reported 70m improvement may partly reflect this leakage.","rationale":"The paper's central claim is an empirical improvement from a human-guided loss, and the only channel through which human data enters training is σ(d,v). Because Psych-ER was sampled before the actor split and includes test actors, σ is contaminated by test-set human responses. This is not a matter of interpretation; it affects whether the reported comparison is a fair, held-out evaluation. A clean rerun is straightforward and would settle it. The reader's other concern about lacking a non-human center-penalty control is real but secondary: it questions attribution to human data, while the leakage issue questions the validity of the improvement itself. If the clean rerun preserves the 70m gain, the missing ablation still weakens the 'human-guided' interpretation, but the empirical result stands; if the gain vanishes, the central result fails. Thus the verdict should remain conditional pending the rerun.","tokens_in":15599,"tokens_out":6020,"duration_ms":59563,"concrete_test":"Recompute mAP@0.00(d,v) in Eq. (2) using only Psych-ER responses on images of the 80 train + 10 validation actors, then retrain the RetinaNet-R101-FPN with the psychophysical loss under the same hyperparameters and five seeds, and evaluate on the same 10 held-out test actors. If the 70m mAP@0.50 improvement over baseline persists within the reported error bars, the leakage concern is resolved; if it shrinks or disappears, the central result is not robust to excluding test-actor human data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 draws the Psych-ER stimulus set as one image per actor per (distance, visibility) cell, and Section 5 then randomly assigns actors to train/val/test splits. Nothing in Section 3.1 excludes the 10 test actors, so the human accuracy surface in Fig. 6a, which enters Eq. (2) as σ(d,v)=100−mAP@0.00(d,v), is computed from responses on images that later appear in the test split. That σ(d,v) sets the width of the Gaussian center penalty in Eq. (4) for every training sample in that distance/visibility cell. The training objective is therefore tuned with statistics that include test-actor images, and the reported mAP@0.50 gain at 70m could be an artifact of this transductive leakage rather than of human-guided training. The paper provides no ablation with σ computed only from the 90 non-test actors, so the magnitude of this effect is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper contributes Psych-ER, a crowdsourced behavioral dataset of human search on aerial NOMAD images, and uses the human accuracy data to define a psychophysical bounding-box regression loss for RetinaNet. The loss adds a Gaussian center penalty whose width is set per (distance, visibility) cell by sigma(d,v)=100-mAP@0.00(d,v) from Eq. (2). The authors train RetinaNet-R101-FPN with and without this loss under an 80/10/10 actor split, with five runs per condition, and report mAP@0.50 improvements at 70m across occlusion levels while not degrading performance at closer distances. They claim this is the first human-guided approach to the localization component of a detection model.","tokens_in":15765,"tokens_out":5635,"duration_ms":53028,"significance":"Psych-ER is a valuable dataset resource, and the paper is explicit about releasing code and data. The loss adaptation is cheap at inference, and the experimental protocol has real strengths: actor-level rather than image-level splits, five repeated runs, and evaluation stratified by distance and visibility. The idea that human accuracy data can inform a localization loss is interesting and, if established, would extend prior human-guided classification work to object detection. However, the current evidence for the headline 70m improvement is weakened by two confounds: sigma is computed from human responses on images of actors later used for testing, and no fixed-width center-penalty baseline is run. These are fixable, but they are load-bearing for the central empirical claim.","major_comments":[{"comment":"Because Section 3.1 samples one Psych-ER image per actor per (distance, visibility) cell before Section 5 assigns the 10 test actors, the human responses used to compute sigma(d,v) include responses to images of the test actors. Eq. (4) uses this sigma to set the width of the Gaussian center penalty for all training samples in that cell, so the training objective is adjusted using statistics that depend on test-actor images. The 70m improvement in Fig. 11b could therefore be an artifact of this transductive leakage. Please recompute sigma excluding the 10 test actors (or use nested splits) and show that the improvement remains; an ablation with sigma estimated only from the 90 non-test actors is needed.","section":"Section 3.1, Section 5, Eq. (2)"},{"comment":"Eq. (4) adds a center-penalty term to the SmoothL1 regression loss, but the paper never compares against a baseline with a fixed-width center penalty. Without such a baseline, one cannot tell whether the reported mAP@0.50 gains come from the human-derived sigma(d,v) or simply from adding any center-location prior to the regression head. Please include a fixed-sigma and/or constant-width Gaussian center-penalty baseline with identical hyperparameters, and report the same distance/visibility breakdown.","section":"Section 5, Eq. (4)"},{"comment":"The manuscript says both scenarios were repeated five times 'to obtain statistical data,' but Fig. 11 presents only mean curves and no confidence intervals or hypothesis tests. The central claim is a per-cell improvement at 70m across occlusion levels; with five runs, paired per-run differences should be shown with intervals or tests to establish that the improvement is not within run-to-run variability.","section":"Section 5, Fig. 11"}],"minor_comments":[{"comment":"Equation (4) contains the typo 'def aultloss'; it should read 'default loss'.","section":"Eq. (4)"},{"comment":"Reference [7] contains a stray '[ ?,' placeholder before '[8, 48, 73]' and should be cleaned.","section":"References"},{"comment":"The text says the 10 positive images per survey had to be of 'different actor'; the wording should clarify whether all 10 actors are distinct or whether a few repeats are allowed.","section":"Section 3.2"},{"comment":"Figure 6's shaded areas are described as standard deviation from a 10-fold partition; the paper should specify what is being partitioned (images, actors, or worker responses).","section":"Fig. 6"},{"comment":"The hyperparameters A and B are fixed to 0.05 and 0.95 without any sensitivity analysis; a sentence justifying the choice or a small ablation would help.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the sigma leakage is real and lands on the abstract's headline claim. The dataset is a solid resource, and the non-test-actor sigma ablation plus a fixed-width center-penalty baseline should be sufficient to test the claim. I would not require new architectures, but I would ask for statistical reporting on the 70m cells and a careful statement of what the mAP@0.50 evaluation does and does not establish about localization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe Psych-ER dataset is the real contribution here. More than 5000 NOMAD images with human accuracy, cursor paths, and response times, collected with care (practice screens, quality rejections, controls) and released on GitHub. That alone is worth serious consideration for the aerial-detection and human-perception communities.\n\nThe loss adaptation is simple: a Gaussian center penalty whose width is set by human accuracy per (distance, visibility) cell. It's a clean idea, and the paper is honest that Eq. (1) borrows from CenterNet. The experiments use RetinaNet-R101-FPN with five runs and an actor split, which is better discipline than much of the field.\n\nBut the experimental support for the paper's main claim—that human guidance improves detection—doesn't currently hold up. Two things are missing.\n\nFirst, there's no control with a fixed-width or non-human center penalty. Without that, the mAP@0.50 improvement at 70m could just be from adding any center loss, not from the human-derived σ. The paper needs this control to attribute anything to psychophysics.\n\nSecond, the stress-test note is correct: Section 3.1 builds Psych-ER with one image per actor per cell, and Section 5 splits actors into train/val/test afterward. Nothing excludes test actors from the human accuracy surface in Fig. 6a. So the σ(d,v) used in training includes responses on test-actor images. The leakage is weak—the test actors contribute roughly 10% of each cell's statistic—but it's real, and no ablation shows how much the 70m gain depends on it. Fix: recompute σ from the 90 non-test actors only, and report.\n\nThere are smaller caveats: one detector, one dataset, 10 epochs, and the mAP@0.50 metric is chosen partly because it matches the loss objective. They do report mAP@0.50:0.95 with no degradation, which is good.\n\nBottom line: this is a solid dataset paper with an under-supported loss claim. It deserves peer review, not desk rejection. The authors should be asked to add the fixed-sigma control and the leak-free σ rerun. If the 70m improvement survives those, the claim will be credible; if not, the dataset still stands on its own.","headline":"Useful new behavioral dataset, but the loss experiment lacks the controls needed to attribute the reported 70 m gain to human data.","tokens_in":16305,"tokens_out":2891,"would_cite":false,"duration_ms":27579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that human search behavior on aerial images can be compressed into a distance- and occlusion-dependent accuracy surface, and that folding this surface into a detector's bounding-box regression loss improves detection of…","keywords":["aerial person detection","search and rescue","visual psychophysics","occlusion","human-guided loss","bounding box regression","crowdsourced behavioral dataset","low-resolution detection"],"falsifier":"Retrain the psychophysical-loss model after recomputing the human accuracy surface with all images of the ten test actors excluded; if mAP@0.50 at 70 m no longer beats the baseline, the reported improvement depends on information from the test set rather than on the psychophysical mechanism.","tokens_in":1501,"feed_emoji":"🚁","tokens_out":1928,"duration_ms":113438,"temperature":0.7,"pith_summary":"The paper tries to show that a person detector for rescue drones can become better at finding occluded people far away if its training loss is shaped by how well humans actually find people in aerial images. Crowd participants searched more than 5,000 images and, when asked to 'find the person,' pointed at the person's location rather than fitting a tight box around them. The authors used that human accuracy to set the strength of a center-distance penalty in the detector's regression loss. Trained this way, a standard one-stage detector improves mAP@0.50 at 70 meters across occlusion levels without degrading performance at closer distances. The paper also claims this is the first human-guided approach aimed at the location component of a detection model rather than its classification component.","feed_headline":"Human search data teach drone AI to spot occluded people at 70m","feed_subtitle":"A human-derived loss improves drone detection of occluded people at long range without hurting close-range accuracy.","key_machinery":"The psychophysical loss is the central mechanism: a center-distance penalty whose width is set by human accuracy at each distance $d$ and visibility $v$ through the identity $\\sigma(d,v)=100-\\mathrm{mAP@0.00}(d,v)$. The human penalty $p$ is blended with the default regression loss by $A\\,p + B(1-p)\\,\\mathcal{L}_{\\mathrm{default}}$, which makes the model concentrate on locating the person's center first and only later on tightening the box. This carries the argument because all reported gains flow from replacing the default loss with this human-modulated version while keeping the architecture, data split, and hyperparameters fixed.","core_discovery":"The central claim is that human accuracy data can guide the localization head of an object detector. From crowd responses on the NOMAD aerial dataset, the authors built a per-distance, per-visibility accuracy surface and turned it into a Gaussian penalty around each ground-truth center with variance $\\sigma(d,v)=100-\\mathrm{mAP@0.00}(d,v)$. The regression loss becomes $A\\,p + B(1-p)\\,\\mathcal{L}_{\\mathrm{default}}$, where $p=1-\\exp(-((x_{\\mathrm{pred}}-x_{\\mathrm{gt}})^2+(y_{\\mathrm{pred}}-y_{\\mathrm{gt}})^2)/(2\\sigma(d,v)^2))$, so human performance controls how much center error is punished: strict where humans find people easily, lenient where humans struggle. On a RetinaNet detector (a one-stage detector) trained on NOMAD, this psychophysical loss improves mAP@0.50 at 70 m across occlusion levels compared with the default regression loss, leaves lower distances unchanged, and does not degrade mAP@0.50:0.95.","pith_inferences":["A useful control would replace the human accuracy surface with a hand-designed sigma schedule that is strict at short range and loose at long range; if the 70 m gain persists, the improvement comes from the loss shape rather than from the human measurements.","The same center-penalty mechanism could transfer to other low-resolution search tasks, such as maritime or post-disaster victim search, where human center-clicks are cheaper to collect than precise bounding boxes.","Reaction-time and search-path data, which the paper collected but did not feed into the loss, could sharpen the sigma surface further, since humans respond faster on true-positive trials."],"forward_implications":["A RetinaNet trained with the psychophysical loss beats the default-loss baseline at 70 m across occlusion levels when measured by mAP@0.50.","The long-range gain does not come at the cost of box tightness: mAP@0.50:0.95 is not degraded relative to the baseline.","The extra cost of the approach is confined to training; inference is unchanged, which matters for deployment on small drones with limited computing.","Psych-ER, with accuracy, response time, and cursor search paths for more than 5,000 images, is a reusable resource for further human-guided detection research.","Under the paper's framing, location accuracy rather than box tightness is what humans actually optimize when searching for people, so mAP@0.50 is the relevant success metric for emergency-response detection."],"supporting_citations":[{"why":"Supplies the aerial images, distance and visibility labels, ground-truth boxes, and the actor-based train/test split that every result depends on.","marker":"[79]"},{"why":"Defines the detector architecture whose bounding-box regression loss the paper modifies.","marker":"[54]"},{"why":"Provides the normal-density center formulation that the psychophysical loss is built on.","marker":"[97]"},{"why":"Official implementation of the detector used for training and testing.","marker":"[92]"},{"why":"Evidence that humans outperform CV models on aerial rescue detection, motivating the use of human behavior as a guide.","marker":"[60]"},{"why":"Establishes the general practice of informing machine perception with psychophysical measurements, which this paper extends to localization.","marker":"[34]"}],"fun_headline_variants":["Human-guided loss sharpens drone detection of occluded people at 70m","Drone AI uses human search data to see occluded people further","Psychophysical loss helps drones spot occluded people at distance","Human data guides drone detector to find occluded people at 70m","Crowd-sourced human perception improves drone detection of occluded persons"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The whole comparison rests on the assumption that the human accuracy surface was measured without leaking the test actors into the loss, and that 'did the human find the person' is the right success measure for rescue detection.","fun_headline_variants_meta":{"raw":{"variants":["Human-guided loss sharpens drone detection of occluded people at 70m","Drone AI uses human search data to see occluded people further","Psychophysical loss helps drones spot occluded people at distance","Human data guides drone detector to find occluded people at 70m","Crowd-sourced human perception improves drone detection of occluded persons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4061,"prompt_tokens":1075,"completion_tokens":2986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":2893}},"tokens_in":691,"tokens_out":2986,"duration_ms":18331,"temperature":1.0,"reasoning_tokens":2893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:35:51.296352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the psychophysical-loss model after recomputing the human accuracy surface with all images of the ten test actors excluded; if mAP@0.50 at 70 m no longer beats the baseline, the reported improvement depends on information from the test set rather than on the psychophysical mechanism.","supporting_citations":[{"cited_title":"Nomad: A natural, occluded, multi-scale aerial dataset, for emergency response scenarios","cited_arxiv_id":null,"evidence_quote":"Supplies the aerial images, distance and visibility labels, ground-truth boxes, and the actor-based train/test split that every result depends on."},{"cited_title":"Detectron2 ModelZoo,","cited_arxiv_id":null,"evidence_quote":"Official implementation of the detector used for training and testing."},{"cited_title":"Region proposal approach for human detec- tion on aerial imagery","cited_arxiv_id":null,"evidence_quote":"Evidence that humans outperform CV models on aerial rescue detection, motivating the use of human behavior as a guide."},{"cited_title":"Informing machine per- ception with psychophysics","cited_arxiv_id":null,"evidence_quote":"Establishes the general practice of informing machine perception with psychophysical measurements, which this paper extends to localization."}],"review_version":1}