{"id":"11d1f327-ba73-4dcd-b72f-cda5b9050f49","arxiv_id":"2508.09909","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The SHREC 2025 relief-pattern track releases 1,000 synthetic meshes and shows the one submitted method barely beats random on the retrieval task.","lead":"This paper releases a SHREC 2025 benchmark of 1,000 synthetic 3D meshes covered with multiple relief patterns and reports the contest results. The only submitted retrieval method barely beat random guessing, confirming that segmenting and retrieving geometric textures remains an open problem.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'no existing algorithms' claim rests on a single partial submission with a flawed evaluation protocol; a supervised baseline trained on the provided 700-mesh set could refute it.","rationale":"The reader's verdict is CONDITIONAL, which I agree with, but my primary concern differs from the reader's stated weakest assumption. The reader focused on synthetic-vs-real transferability: whether procedurally textured meshes represent real scanned relief patterns. That is a valid external-validity concern, but the paper's strongest claim is about the track's dataset itself ('under the general conditions imposed by the dataset'). The more immediate, load-bearing weakness is that the evidence for the universal negative is a single partial submission with evaluation-protocol flaws. Even if the synthetic dataset were a perfect proxy for real relief patterns, the paper would still not have shown that 'no existing algorithms are capable'—it would only show that the one submitted method and one incomplete attempt failed. The training set with 700 labeled meshes enables a concrete counterfactual test: a straightforward supervised baseline could demonstrate whether the tasks are actually solvable. Because the paper's contribution (the benchmark) is still valuable and the authors are honest about the poor results, a CONDITIONAL verdict (requiring a softened claim or baseline experiments) is appropriate, matching the reader's verdict. Thus, no change in verdict is needed; the concern reinforces the conditionality.","tokens_in":13192,"tokens_out":5872,"duration_ms":64845,"concrete_test":"Train a standard supervised per-face segmentation/retrieval baseline on the 700 training meshes using the provided per-face labels (e.g., a PointNet++ or a geometric-descriptor MLP). Evaluate it on the 54 query meshes for segmentation and on the 300 retrieval meshes for retrieval, using the same metric definitions and query-exclusion rule as in Table 1 (reporting how many queries are excluded). If the baseline achieves AUC or mAP substantially above OMVMLP's 0.523/0.218, or high segmentation accuracy, the central claim is refuted; if it performs near random, the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in Section 1, is that 'no existing algorithms are capable of effectively performing segmentation or retrieval under the general conditions imposed by the dataset.' The only quantitative evidence is the OMVMLP submission (Section 3.2), which achieved AUC 0.523 and mAP 0.218. This evidence is insufficient and partly invalid: (i) OMVMLP's Strategy 2 fine-tunes the classifier on the query set (Section 3.2.1), a transductive/leakage practice that makes the evaluation not a clean test of generalization; (ii) the reported metrics pool two runs and exclude queries without relevant meshes, but the number of excluded queries is never stated (Section 3.2.2), so the effective test set may be small and unrepresentative; (iii) only one partial submission was received from five registrations—an absence of participants is not evidence that no algorithm can solve the tasks. The 700-model training set with per-face ground truth (Section 2) could support a simple supervised baseline; if such a baseline succeeds, the universal negative is false. As written, the open-problem claim goes beyond what the track results demonstrate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on the SHREC 2025 track on retrieval and segmentation of multiple relief patterns on 3D meshes. The track provides a synthetic benchmark of about 1,054 meshes split into a query set (54), a retrieval set (300), and a training set (700 with per-face ground-truth labels), and defines two tasks: retrieving all meshes sharing at least one relief pattern with a query, and segmenting surfaces containing multiple relief patterns. Five groups registered; one group (OMVMLP) submitted partial retrieval results, and one group (KU-3DSeg) provided preliminary qualitative feedback only. The reported OMVMLP performance is near chance (AUC 0.523, mAP 0.218). From this outcome, the paper concludes that no existing algorithms are capable of effectively performing segmentation or retrieval under the conditions of the dataset, and argues that the benchmark is therefore a valuable open-problem testbed.","tokens_in":13480,"tokens_out":4389,"duration_ms":56155,"significance":"If the benchmark is properly validated and the claims appropriately scoped, this is a useful community resource: it is among the largest relief-pattern datasets with automatic per-face ground truth, and the two-task design (segmentation plus retrieval) addresses a realistic gap in existing SHREC tracks. The paper clearly documents dataset construction, task definitions, and evaluation metrics, and the official webpage provides reproducibility resources. However, the paper's central generalization claim is not supported by the evidence it reports, and the only quantitative submission contains a transductive evaluation step. The benchmark could still be a significant contribution after the claims are revised and baseline experiments are added, but in its current form the paper overstates what the track results demonstrate.","major_comments":[{"comment":"The claim that 'no existing algorithms are capable of effectively performing segmentation or retrieval under the general conditions imposed by the dataset' is load-bearing but unsupported. The evidence is one partial submission (OMVMLP, Table 1) plus non-submission by four registered groups. Non-participation does not test any algorithm, and a single near-random result cannot establish a universal negative about existing algorithms. Since the track provides a 700-mesh training set with per-face ground truth (Section 2), a straightforward supervised baseline trained on those labels is the obvious missing control; without it, the paper's central conclusion should be weakened to 'no submitted method performed satisfactorily on this benchmark' or supported by additional baseline experiments.","section":"Section 1 and Section 7"},{"comment":"The quantitative evaluation is not a clean generalization test. In Strategy 2, the classifier is fine-tuned on the query set and then evaluated on the same queries' retrieval relevance labels; this is a transductive/leakage step that inflates or at least obscures the meaning of the reported scores. In addition, Table 1 pools Strategies 1 and 2 and says queries with no relevant retrieval mesh were excluded, but the number of excluded queries is not stated. Without a per-strategy breakdown and the exclusion count, the effective test set size and the near-random AUC cannot be properly interpreted. The authors should report these details or recompute the metrics with the leakage removed.","section":"Sections 3.2.1 and 3.2.2"},{"comment":"KU-3DSeg is described as a proposed method, but no quantitative evaluation is provided, only illustrative segmented outputs. If this method is used to support the claim of task difficulty, the absence of any metric or evaluation protocol is a critical gap. If it is meant only as preliminary feedback, the paper should say so explicitly and avoid presenting it as evidence about the performance of existing algorithms.","section":"Section 3.1 and Figures 7-8"}],"minor_comments":[{"comment":"The caption says 'A single ROC curve is sufficient to evaluate both runs,' but the text says the two strategies differ on the query classes with additional patterns. Unless the ROC curve is identical by construction, the paper should explain why one curve represents both runs, and the reference to 'Fig. 1' in the caption appears to be incorrect.","section":"Section 3.2.2 / Figure 6"},{"comment":"Table 1 would benefit from reporting the number of queries remaining after exclusion, the number of relevant models per query, and separate columns for Strategy 1 and Strategy 2. This is needed for the reader to assess the statistical weight of AUC 0.523.","section":"Section 3.2.2"},{"comment":"The paper says the full dataset consists of 'approximately 1,000 triangular meshes'; the sum of query, retrieval, and training sets is 1,054. It would be clearer to state the exact total or describe the sets as approximately 1,050.","section":"Section 2"},{"comment":"Minor typos: 'adjaceny graph' should be 'adjacency graph'; in Section 4, 'descritptions' should be 'descriptions.'","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and track organization are potentially valuable, and the paper contains a clear description of the benchmark. My main concern is the mismatch between the universal claim about algorithm capability and the thin, partly leaked empirical evidence. The manuscript can be made publishable by narrowing the claim and adding a supervised baseline or another quantitative reference point. I do not see a problem with novelty disclosure; the issue is strictly about evidence and framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a useful SHREC benchmark report, but the headline claim that no existing algorithms can handle the tasks is not supported by the evidence. The dataset and track design are genuinely new: 1,000 meshes (700 with per-face training labels), multiple coexisting relief patterns on a single surface, quasi-regular textures, and disjoint train/retrieval pattern classes. That's a real step beyond the single-pattern SHREC tracks, and the report is transparent about the poor participation and near-random results. The discussion of why image-based projection methods fail—occlusions, varying pattern scale, and the difficulty of localizing multiple textures—is sensible.\n\nThe soft spot is the overreach. The only quantitative evidence is one partial submission (OMVMLP), and that evaluation has problems: Strategy 2 fine-tunes the classifier on the query set, which is leakage; the reported metrics pool both strategies without reporting them separately; and the number of queries excluded for having no relevant retrieval mesh is never stated, so the effective test set may be small. More importantly, the 700-model training set with per-face ground truth is never used for a supervised baseline. A simple learned descriptor could plausibly do far better than AUC 0.523, which would undercut the 'still open' conclusion. The synthetic-to-real transfer question is also worth flagging: the synthetic argument is sound for control, but the generalization to scanned artifacts is not demonstrated.\n\nI'd send this to a serious reviewer. The benchmark itself is a contribution worth having, and the report is honest enough to be salvageable. Ask the authors to temper the 'no existing algorithms' wording, clarify the evaluation protocol, and consider adding a straightforward supervised baseline. I'd bring it to a group working on 3D geometry/texture, but it's too niche for a general reading group.","headline":"Useful new benchmark, but the 'no existing algorithms' claim outruns the evidence.","tokens_in":13994,"tokens_out":2682,"would_cite":true,"duration_ms":28863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68U05","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"This benchmark report claims that segmenting and retrieving multiple relief patterns on 3D meshes is still beyond current algorithms, since no submitted method solved either task on its 1,000-model synthetic dataset.","keywords":["relief patterns","geometric textures","3D mesh segmentation","3D surface retrieval","benchmark dataset","per-face ground truth","synthetic meshes","unseen-class generalization"],"falsifier":"One experiment would settle the central claim: train a per-face classifier on the 700-model training set using only local geometric features (curvature, normals, shape index over several radii) and evaluate per-face label agreement on the 54 query meshes. High accuracy on the 14 single-pattern queries together with near-chance accuracy on the 40 two-pattern queries would show the open problem is specifically pattern boundaries and multi-pattern disambiguation; near-chance accuracy everywhere would confirm the paper's broader claim. Strong accuracy on all 54 queries under the track protocol wou","tokens_in":13136,"feed_emoji":"🏺","tokens_out":10876,"duration_ms":108556,"temperature":0.7,"pith_summary":"This paper reports the outcome of a 2025 contest track built around one question: can an algorithm find and isolate repeated geometric textures (\"relief patterns\") on the surface of a 3D mesh, and then retrieve other meshes carrying the same texture? The track's answer is no, not yet. Across roughly 1,000 procedurally textured synthetic meshes, some carrying two or three different patterns, some with patterns covering only part of the surface, and with training and test pattern classes deliberately disjoint, no submitted method completed either the segmentation or the retrieval task at useful accuracy; the strongest attempt, a multi-view image classifier, reached an AUC of 0.523, essentially random. The paper's contribution is therefore the benchmark itself, with automatic per-face ground truth, plus a diagnosis of why current techniques fail: they are built for single whole-surface patterns, rely on 2D projections that miss occluded textures, and use fixed-radius neighborhoods that blur the boundary between adjacent patterns. If the paper is right, relief pattern analysis is a genuinely open problem with direct applications in artifact restoration, industrial surface inspection, and biomedical imaging.","feed_headline":"No entrant solved the new 3D relief-pattern test","feed_subtitle":"The best try scored barely above chance (AUC 0.52); the 1,000-mesh benchmark is now public.","key_machinery":"The load-bearing object is the dataset itself: roughly 1,000 synthetically generated triangle meshes with procedurally applied relief patterns and automatic per-face ground truth, organized into training (700), retrieval (300), and query (54) sets with deliberately disjoint pattern classes. A relief pattern is defined as a local, repeated deformation on a 3D surface, independent of the object's global shape; the benchmark operationalizes this by applying curated 2D textures as geometric displacements onto base meshes with varied topology. The dataset carries the argument because the track's negative result, that no submitted algorithm solved either task, is meaningful only if the data is a f","core_discovery":"The paper's central claim is that relief pattern analysis remains unsolved under general conditions: surfaces carrying one to three patterns, covering part or all of the surface, with quasi-regular organic textures, on complex meshes with self-occlusions. To test this, the organizers generated roughly 1,000 synthetic triangle meshes by procedurally applying curated 2D textures to base models, producing per-face ground truth automatically, and split the texture classes so the 700 training meshes share no classes with the 300 retrieval meshes or the 54 queries. Five groups registered; one submitted a partial retrieval-only solution (a multi-view image classifier, AUC 0.523), and one documented","pith_inferences":["I read the near-chance AUC as evidence that the discriminative signal lives in local surface geometry, not whole-object appearance; a cheap test would be a random forest over per-face curvature and shape-index histograms trained on the 700-model set. If it beats the submitted image classifier on single-pattern queries alone, the bottleneck is specifically pattern boundaries and unseen classes, not","The synthetic-to-real gap cuts both ways: procedurally textured, noise-free meshes may be easier than scanned artifacts, or real acquisition noise may make the disjoint-class generalization even harder. A small validation set of photogrammetry-scanned engraved objects would calibrate the benchmark's difficulty against the applications the paper cites.","The track's failure rate (one partial submission out of five registrations) mirrors an earlier multi-pattern track, suggesting the bottleneck is a structural mismatch between available methods and the multi-pattern, partially covered setting rather than this dataset's idiosyncrasies; a method that predicts per-face pattern class with scale-adaptive neighborhoods would be the natural next entry."],"forward_implications":["The released 700-model training set, one of the largest for relief pattern analysis, gives future methods a standardized place to train and be compared; the track's protocol (query set, retrieval set, per-face ground truth) becomes the measuring stick.","Projection-based methods that render a few whole-object views cannot reliably solve the task: patterns hidden inside closed or self-occluding geometry are invisible to them, which explains the near-chance AUC of the one submitted retrieval solution.","Retrieval techniques from earlier single-pattern tracks do not transfer: they aggregate features over one representative planar patch, which presupposes a single whole-surface texture, so extending them to multi-pattern, partially covered surfaces requires a different localization step.","Segmentation at pattern interfaces is a concrete bottleneck: fixed-radius neighborhoods spanning two adjacent patterns produce ambiguous per-face features, so any viable method must handle boundaries explicitly or adapt its receptive field.","Because texture classes are disjoint between training and retrieval/query sets, learning-based methods must recognize previously unseen patterns, so the benchmark tests generalization rather than memorization."],"supporting_citations":[{"why":"The prior single-pattern geometric-relief retrieval track whose methods and evaluation metrics (NN, FT, ST, mAP, nDCG, e, AUC) this track inherits and compares against.","marker":"[29]"},{"why":"The earlier multi-pattern recognition track whose low completion rate and difficulty this track's outcome mirrors.","marker":"[3]"},{"why":"The earliest track on retrieving surface patches with similar reliefs, establishing the problem lineage and the single-pattern dataset convention.","marker":"[2]"},{"why":"Supplies the definition of a relief pattern as a local, repeated deformation independent of the object's global structure.","marker":"[17]"},{"why":"The pottery-retrieval convolutional network whose multi-view, image-based design the submitted retrieval method adapts.","marker":"[1]"},{"why":"The unsupervised 3D texture segmentation method whose local geometric feature extraction and UV-mapping pipeline the per-face submission builds on.","marker":"[12]"},{"why":"Prior work on binary relief-pattern segmentation cited for the known difficulty of patterns with very different spatial scales.","marker":"[32]"},{"why":"The curated texture collection from which the track's 14 relief pattern categories are selected for dataset generation.","marker":"[33]"}],"fun_headline_variants":["No entrant cracked the 3D relief-pattern test","Relief-pattern segmentation: all teams fail to beat chance","SHREC '25 relief track: best AUC 0.52, problem open","3D relief patterns still unsolved after SHREC 2025"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The report's conclusion that relief pattern analysis is an open problem assumes that procedurally textured synthetic meshes, with their noise-free surfaces and automatic labels, are a faithful proxy for real surfaces carrying reliefs; if real scanned surfaces behave differently, the benchmark's difficulty may not transfer to the applications the paper cites.","fun_headline_variants_meta":{"raw":{"variants":["No entrant cracked the 3D relief-pattern test","Relief-pattern segmentation: all teams fail to beat chance","SHREC '25 relief track: best AUC 0.52, problem open","3D relief patterns still unsolved after SHREC 2025"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1096,"prompt_tokens":619,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":363,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":363,"tokens_out":477,"duration_ms":5539,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:42:56.587715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One experiment would settle the central claim: train a per-face classifier on the 700-model training set using only local geometric features (curvature, normals, shape index over several radii) and evaluate per-face label agreement on the 54 query meshes. High accuracy on the 14 single-pattern queries together with near-chance accuracy on the 40 two-pattern queries would show the open problem is specifically pattern boundaries and multi-pattern disambiguation; near-chance accuracy everywhere would confirm the paper's broader claim. Strong accuracy on all 54 queries under the track protocol wou","supporting_citations":[{"cited_title":"SHREC 2020: Retrieval of digital surfaces with similar geomet- ric reliefs","cited_arxiv_id":null,"evidence_quote":"The prior single-pattern geometric-relief retrieval track whose methods and evaluation metrics (NN, FT, ST, mAP, nDCG, e, AUC) this track inherits and compares against."},{"cited_title":"SHREC’18 track: Recognition of geometric patterns over 3D models","cited_arxiv_id":null,"evidence_quote":"The earlier multi-pattern recognition track whose low completion rate and difficulty this track's outcome mirrors."},{"cited_title":"SHREC’17 Track: Retrieval of Surfaces With Similar Relief Patterns","cited_arxiv_id":null,"evidence_quote":"The earliest track on retrieving surface patches with similar reliefs, establishing the problem lineage and the single-pattern dataset convention."},{"cited_title":"Effective characterization of relief patterns","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of a relief pattern as a local, repeated deformation independent of the object's global structure."},{"cited_title":"Convolutional neural network for pottery retrieval","cited_arxiv_id":null,"evidence_quote":"The pottery-retrieval convolutional network whose multi-view, image-based design the submitted retrieval method adapts."},{"cited_title":"3D-TexSeg: Unsupervised Segmentation of 3D Texture using Mutual Transformer Learning","cited_arxiv_id":"2311.10651","evidence_quote":"The unsupervised 3D texture segmentation method whose local geometric feature extraction and UV-mapping pipeline the per-face submission builds on."},{"cited_title":"Binary segmentation of relief patterns on point clouds","cited_arxiv_id":null,"evidence_quote":"Prior work on binary relief-pattern segmentation cited for the known difficulty of patterns with very different spatial scales."},{"cited_title":"CC0 PBR Textures","cited_arxiv_id":null,"evidence_quote":"The curated texture collection from which the track's 14 relief pattern categories are selected for dataset generation."}],"review_version":1}