{"id":"04e7839f-238c-4a39-91e0-d607165870bc","arxiv_id":"2411.18776","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A leaf-shaped occlusion can flip some traffic sign classifications in the LISA-CNN model, but the evidence is limited to five signs and best-case placements.","lead":"This paper proposes a new adversarial attack that places images of fallen leaves on traffic signs to trick a convolutional neural network into misclassifying the signs. The authors report misclassifications on a small set of test images and analyze edge detection metrics as a possible defense cue.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-case grid-search selection inflates reported attack success; the paper reports the maximum over placement, patch ratio, and rotation rather than the success rate of naturally occurring leaf placements.","rationale":"The reader's weakest assumption correctly identifies the core problem: the grid search chooses the best placement for each test image and the paper reports that best result as the attack's success rate. My independent reading reaches the same conclusion. The paper's stated threat model depends on leaves appearing natural and incidental, so the evaluation should sample the distribution of natural leaf positions, sizes, and rotations; instead it reports a maximization. This is not an internal inconsistency in the method, but it is a load-bearing validity gap for the central claim of a high success rate. The 97% confidence statement is also misleading because it comes from the strongest Merge rows, not from the average successful attack. I do not find the concern to be refuted by the edge-detection analysis, since that analysis is post hoc and uses the same selected images. The existing evidence supports a demonstration that some leaf placements can cause misclassification, but it does not support the practical, high-success-rate claim. Therefore I keep the reader's REJECT verdict; since my stress-test does not change that verdict, I mark the outcome UNCHANGED.","tokens_in":11049,"tokens_out":2530,"duration_ms":26267,"concrete_test":"Run a randomized control evaluation: for each of the same five signs and three leaf types, draw 1,000 placements uniformly from positions where the leaf fits inside the sign mask, with rotation uniform over [0°, 360°) and patch ratio uniform over [0.1, 0.5]. Record the fraction of natural placements that cause LISA-CNN to misclassify and the median confidence of successful attacks. Compare with Table 1's 10/15. If the random success rate is near chance, or below roughly 30%, the grid-search result is not representative of naturally occurring leaf falls. Also report success per sign rather than per image–leaf combination, with a confidence interval, to quantify the small-sample variability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a high success rate rests on Table 1, where each of the 15 image–leaf pairs is represented by the single placement, patch ratio, and rotation that the grid search in Section 4.1.1 found best. That is an upper envelope over the search space, not an estimate of real-world attack effectiveness. Because the proposed attack is explicitly a natural artifact, the relevant quantity is the success probability over the distribution of natural leaf falls, not the maximum over positions, rotations, and sizes. The search maximizes exactly the quantity reported, so selection bias is built into the headline number: 10/15 successful image–leaf combinations does not estimate the chance that a random leaf on a random sign fools LISA-CNN. The sample is also only five signs, one image per sign, and three leaves, so the success count has wide variability. The 97% confidence cited in the abstract is the best case among the already-selected Merge rows in Table 1, not a typical successful attack. The edge-detection correlation in Section 6 is post hoc and computed on the same selected adversarial images, so it does not independently support practical significance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new class of adversarial attacks on traffic sign classifiers in which fallen tree leaves are placed on signs to induce misclassification. The attack is evaluated on five LISA test images (Yield, Stop, Turn Right, Pedestrian Crossing, Merge) using three leaf types (Maple, Oak, Poplar). For each image–leaf combination, a grid search over leaf position, patch ratio, and rotation selects the configuration with the highest misclassification confidence. Table 1 reports 10 successful misclassifications out of 15 combinations. The paper also measures Canny edge-detection metrics on the original and adversarial images and compares averages for successful versus unsuccessful attacks. The central claim is a high attack success rate with misclassified-image confidences of 97%, and that edge-detection metrics correlate with attack success.","tokens_in":11276,"tokens_out":2166,"duration_ms":20927,"significance":"The idea of using naturally plausible, deniable artifacts such as leaves as physical adversarial perturbations is interesting and thematically extends prior work on shadows and natural light. A rigorous study of this phenomenon could be valuable for the trustworthy-autonomy community. However, the evidence presented here is far too limited and methodologically skewed to support the claimed high success rate or the edge-detection correlation. The paper's contribution is currently at the level of a preliminary observation rather than a demonstrated attack class.","major_comments":[{"comment":"The attack success rate is defined as the best result of a grid search over leaf position, patch ratio, and rotation for each image–leaf pair. The paper reports 10/15 'successful' combinations, but this is an upper envelope over the search space, not the success probability of a naturally placed leaf. Because the threat model in §3 is explicitly a real-world leaf falling and sticking to a sign, the relevant quantity is the success rate over the distribution of naturally occurring leaf placements. Reporting the maximum over searched configurations as the attack success rate introduces selection bias into the headline number and overstates practical effectiveness.","section":"§4.1.1 and Table 1"},{"comment":"The experimental basis is only five test images (one per sign type) and three leaf images. Table 1 contains 15 data points, and the 10/15 count has a wide confidence interval; it cannot support a general claim of 'high success rate' for traffic sign classification. Moreover, the five signs are not a representative sample of the 16-class LISA-CNN task, and no statistics over multiple images per sign type are provided.","section":"§5 and Table 1"},{"comment":"The abstract and Section 1 state that 'misclassified images having confidence values of 97%,' but this is true only for the three Merge rows in Table 1 (96.67–97.21%). Other successful attacks have substantially lower confidence, e.g., Stop Maple at 59.23% and Turn Right Poplar at 37.62%. The 97% figure is the best case among already-selected adversarial images, not a representative or aggregate confidence.","section":"Abstract and §5"},{"comment":"The edge-detection correlation analysis is post hoc and uses the same grid-search-selected adversarial images that define success. The comparison of averages between successful and unsuccessful attacks in Table 3 is not accompanied by any statistical test, confidence intervals, or control for confounds such as sign type and leaf type. Therefore the conclusion that 'analyzing edge detection metrics could have benefit in detecting adversarial attacks' is unsupported by the data.","section":"§6 and Table 3"}],"minor_comments":[{"comment":"The text refers to 'Polar' leaves, but the intended species name is 'Poplar' as used in Table 1 and Figure 3; this inconsistency appears several times and should be corrected.","section":"§4.2"},{"comment":"The caption lists '(s) Turn Right Poplar' twice and '(t) Yield Poplar' once, but the figure appears to show the same image twice; the labeling should be fixed to match the displayed images.","section":"Figure 5 caption"},{"comment":"There is a typo in 'an malicious human attacker' which should be 'a malicious human attacker.'","section":"Abstract"},{"comment":"The paper states that varying rotation angles will 'uncover whether specific orientations contribute more effectively to misclassification,' but no analysis of rotation effects is presented.","section":"§4.1.3"},{"comment":"The center-of-gravity distance change values are reported without units or explanation of the spatial scale, making the averages hard to interpret.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"The core idea is worth one solid experiment, but the current manuscript would require new data collection (many more images per sign type, multiple natural-placement distributions, and a proper success-rate estimator) to fix the load-bearing selection-bias and sample-size problems. That is beyond a routine revision. The paper also does not compare against existing occlusion-based baselines or report a query budget for the black-box grid search, which weakens its positioning relative to prior physical adversarial attacks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is using leaves as adversarial occlusions on traffic signs. That's a real increment over stickers, shadows, and light attacks, and the plausible-deniability angle is sensible: a leaf on a sign looks like an accident, not an attack. The paper also does a reasonable job of building leaf masks and varying size, rotation, and leaf type, and it is clear about the threat model being black-box. I'll credit the authors for a clean framing and for acknowledging in Section 5 that some sign–leaf combinations failed.\n\nThe problem is the central claim. The 'high success rate' (10/15 combinations) comes from Table 1, where each row is the single best placement, patch ratio, and rotation found by the grid search described in Section 4.1.1. That is an upper envelope over the search space, not an estimate of how often a naturally falling leaf would fool LISA-CNN. The paper even says the grid search 'identifies the combination ... that yields the highest confidence' — so the reported numbers are exactly the maximum. For an attack whose selling point is natural plausibility, the relevant statistic is the success probability over natural leaf placements, and this paper does not provide it. The 97% figure in the abstract is also cherry-picked: it is the best confidence among the three Merge rows, not a typical successful attack; other successes have confidence in the 37–82% range.\n\nThe edge detection analysis in Section 6 is post hoc. It compares metrics of successful versus unsuccessful attacks on the same selected adversarial images, with only 10 successes and 5 failures, and without any independent validation or statistical test. The observed correlations may be real, but they are not demonstrated. The sample size is also tiny: five signs, one image each, three leaf types. No baseline like random placement or unoptimized placement is given, so we cannot tell whether the grid search is merely finding rare fragile spots or exposing a systematic vulnerability.\n\nThat said, the idea is worth a careful follow-up. With a proper evaluation over many sign instances, a distribution of leaf placements, and a comparison against a random-placement baseline, this could become a solid paper. As it stands, the evidence is too thin to support the abstract's claims, and the paper probably needs major revision rather than acceptance.\n\nWho is this for? Researchers working on physical adversarial attacks and traffic sign robustness. It deserves a serious referee because the attack vector is novel and the limitations are fixable, but I would expect a reject-and-resubmit outcome.","headline":"Leaves as a physically plausible occlusion attack is a small new idea, but the reported success rate is the best case of a grid search and overstates what the evidence supports.","tokens_in":11749,"tokens_out":1036,"would_cite":false,"duration_ms":12242,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fallen leaves fool traffic-sign AI in 10 of 15 tests.","keywords":["adversarial attack","traffic sign classification","leaf occlusion","physical-world attack","plausible deniability","LISA dataset","LISA-CNN","edge detection"],"falsifier":"Take a set of traffic-sign images, drop leaves on them using a physical or simulated model of natural leaf fall (random position, angle, and size from realistic distributions), and measure LISA-CNN's misclassification rate; if that rate is close to zero while the grid-search-optimized rate remains 10 of 15, the central claim is not supported for natural conditions.","tokens_in":10872,"feed_emoji":"🍂","tokens_out":6130,"duration_ms":54257,"temperature":0.7,"pith_summary":"This paper seeks to establish that nature-made objects, specifically single tree leaves placed on traffic signs, form a new class of physical adversarial perturbations. In tests on five real signs from the LISA dataset, the LISA-CNN classifier misclassified 10 of 15 leaf-sign combinations after a grid search picked the leaf's position, size, and rotation, with the strongest misread reaching about 97 percent confidence. Because a leaf on a sign could plausibly have fallen from a nearby tree, the attack carries deniability that stickers, paint, or flashlights lack. The paper also argues that successful attacks shift edge-detection statistics more than unsuccessful ones, which could help future defenses. If correct, the work broadens the physical attack surface for autonomous-vehicle perception from human-made artifacts to ordinary seasonal litter.","feed_headline":"Fallen leaves fool traffic-sign AI in 10 of 15 tests","feed_subtitle":"Real maple, oak, and poplar leaves trigger misreads; the strongest confidence hits 97 percent.","key_machinery":"The load-bearing mechanism is an optimized leaf-mask overlay. Each leaf image is converted to a binary mask by grayscale conversion, Gaussian blur, Canny edge detection, dilation, morphological closing, and contour selection; the mask is then placed on a traffic sign within the sign's binary mask, and a grid search over candidate positions, patch-area ratios from 0.1 to 0.5, and eight rotation angles selects the placement with the highest confidence in the wrong label. The edge-analysis part computes Canny edge length, orientation from Sobel gradients, intensity, and center of gravity for the attacked images. The grid search is what turns a random-looking occlusion into an effective perturbation, and the edge metrics provide the paper's proposed signal for distinguishing successful from unsuccessful attacks.","core_discovery":"The central claim is that ordinary autumn leaves can serve as adversarial perturbations against a traffic-sign classifier. The paper overlays binary masks of Maple, Oak, and Poplar leaves onto five LISA sign images (Yield, Stop, Turn Right, Pedestrian Crossing, Merge), each at a position, scale, and rotation chosen by grid search to maximize misclassification confidence. The result is that 10 of the 15 sign-leaf combinations are misclassified, and the Merge sign is read as Pedestrian Crossing with 96-97 percent confidence for all three leaf types, while the Yield sign resists all three. Looking at Canny edge-detection metrics, the paper reports that successful attacks show larger relative changes in edge orientation and intensity than unsuccessful attacks. The contribution is the new attack class and the plausible deniability nature-made occlusions provide.","pith_inferences":["The grid-search optimum is an upper bound on attack success; a field test with leaf positions sampled from natural wind and gravity distributions would show how often real leaves land close enough to the optimum to fool the classifier.","The five test images are a small sample, so the per-sign pattern (Merge always fooled, Yield never) is not evidence about the population of U.S. traffic signs; broader sampling is needed before \"high success rate\" is taken as a general property.","The edge-statistics correlation is computed on the same 15 attacked images; turning it into a reliable detector would require separate data with natural leaf occlusions and unrelated obstructions.","An attacker in the physical world would have to approximate the searched position, scale, and rotation by hand or by waiting for favorable leaf falls; the paper does not demonstrate that such placements are physically achievable, so a manual or robotic placement study is the obvious next experiment."],"forward_implications":["If the central claim is right, physical adversarial attacks can be made from unmodified, naturally occurring objects instead of manufactured patches or stickers.","A vehicle's traffic-sign classifier can be seriously wrong about signs such as Merge, Pedestrian Crossing, and Turn Right when a single leaf occludes part of the sign.","Nature-based occlusions weaken the attribution of a misclassification to an attacker, since a leaf on a sign is an everyday occurrence.","The reported edge-statistics differences suggest that edge-based features, especially intensity and orientation changes, could be used to detect this kind of occlusion attack."],"supporting_citations":[{"why":"Supplies the LISA-CNN classifier and the RP2 physical-attack baseline this leaf attack builds on.","marker":"[7]"},{"why":"Establishes the preceding natural-phenomenon attack (natural light) that this work extends to leaves.","marker":"[12]"},{"why":"Provides the grid-search optimization method used to choose leaf position, patch ratio, and rotation.","marker":"[18]"},{"why":"Supplies the LISA dataset and the 16-sign subset from which the five test images come.","marker":"[25]"},{"why":"Gives the Canny edge-detection algorithm whose output metrics drive the paper's correlation analysis.","marker":"[32]"},{"why":"Makes the foundational demonstration that deep networks are susceptible to adversarial inputs, motivating the whole threat model.","marker":"[36]"}],"fun_headline_variants":["Autumn leaves trick sign-classifying AI in 10 of 15 trials","Nature's own attack: leaves mislead traffic sign AI","Merge sign reads as Pedestrian Crossing with 97% confidence when leaf-covered","Leaves offer plausible deniability for adversarial sign attacks","Maple, oak, poplar: real leaves foil traffic sign AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack's reported success rate depends on the grid search being allowed to choose the best leaf position, size, and rotation for each image; if naturally fallen leaves rarely land in those optimized placements, the real-world success rate could be much lower.","fun_headline_variants_meta":{"raw":{"variants":["Autumn leaves trick sign-classifying AI in 10 of 15 trials","Nature's own attack: leaves mislead traffic sign AI","Merge sign reads as Pedestrian Crossing with 97% confidence when leaf-covered","Leaves offer plausible deniability for adversarial sign attacks","Maple, oak, poplar: real leaves foil traffic sign AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2700,"prompt_tokens":956,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1653}},"tokens_in":572,"tokens_out":1744,"duration_ms":15357,"temperature":1.0,"reasoning_tokens":1653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:53:00.734057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of traffic-sign images, drop leaves on them using a physical or simulated model of natural leaf fall (random position, angle, and size from realistic distributions), and measure LISA-CNN's misclassification rate; if that rate is close to zero while the grid-search-optimized rate remains 10 of 15, the central claim is not supported for natural conditions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the preceding natural-phenomenon attack (natural light) that this work extends to leaves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LISA dataset and the 16-sign subset from which the five test images come."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Canny edge-detection algorithm whose output metrics drive the paper's correlation analysis."}],"review_version":1}