{"id":"8ad12555-c9b5-43da-982b-1f25b70a6e49","arxiv_id":"2508.12671","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A non-interpretable rarity meter built by minimizing a weighted scaling stress on NFT trade dissimilarities outperforms existing meters under a new performance measure that it was designed to optimize.","lead":"This paper proposes a new way to rank NFTs by rarity, treating the ranking as a one-dimensional map fitted to trade prices, and introduces a new performance measure called DIT. The resulting black-box rarity score beats existing meters on the authors' benchmark, but the benchmark is scored with the same measure that their method is designed to optimize.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority of DIT is evaluated under the same objective F that DIT is trained to minimize, and F is never validated against the established weighted correlation; the head-to-head result is not independent evidence.","rationale":"The paper's central empirical claim is that DIT demonstrates superior performance compared to existing rarity meters. For that claim to hold, the performance measure used in the comparison must be a valid, externally meaningful criterion for rarity-meter quality. The manuscript adopts a new measure F in Section 7.2, replacing the established weighted correlation Fwc, without validating that F agrees with Fwc or with any other external outcome. Since DIT is constructed by minimizing the same stress-based objective F, evaluating competitors under F gives DIT an intrinsic advantage: interpretable meters are only rescaled by the optimal scalar alpha, while DIT can freely choose the entire rarity ordering to reduce F. This is not a fully circular evaluation because DIT's test scores are produced by out-of-sample smoothing on a held-out split, but the metric itself remains the training objective of the proposed method. The paper itself flags the substitution, making this the obvious load-bearing assumption. I also note that the theoretical 'global optimality' claims are overstated: Theorem 1 guarantees convergence to a stationary point of the smoothed stress, and the continuation procedure in Section 5 does not by itself ensure a global minimum of the unsmoothed S. However, the empirical claim is the main contribution, and it rests on the unvalidated F. The reader's weakest assumption identifies exactly this issue. A single re-run under Fwc would settle whether the superiority is real or an artifact, so the concern is concrete and testable. Given that the current evidence is insufficient to support the headline claim, the REJECT verdict is appropriate.","tokens_in":14710,"tokens_out":3150,"duration_ms":36871,"concrete_test":"Recompute the performance profiles of Figure 2 using the established weighted correlation Fwc (Section 3.1) on the same 100 collections, the same train/test split, and the same scale-adjustment protocol. If DIT does not rank first or statistically tie with the best interpretable meter under Fwc, the claimed superiority is an artifact of evaluating with DIT's own objective.","verdict_should_be":"REJECT","load_bearing_attack":"Section 7.2 explicitly replaces the established ROAR measure Fwc with a new measure F, which is the NWUDS stress that the proposed DIT meter is designed to minimize. The performance profiles in Figure 2 are then computed under F, and the comparison meters are only scale-adjusted to F via Statement 1. Because DIT directly optimizes F on the training data, while the competitors are not optimized for F, the claim that 'DIT consistently outperforms the other methods' (Section 7.3) is largely an artifact of evaluating each method by DIT's own objective, not by market alignment. No evidence is given that F ranks meters similarly to Fwc or predicts external outcomes; the paper simply asserts the replacement on computational grounds in Section 7.2. The out-of-sample smoothing in Section 6.2 softens the circularity, but the evaluation metric is still the same loss family that DIT minimizes. Without validation of F, the superiority claim is not a falsifiable empirical result about NFT rarity, but a statement about the objective DIT was built to optimize. This is the load-bearing weakness: if the established weighted correlation were used instead, the headline result could disappear.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dimension-reduction formulation of NFT rarity. Given trade-derived pairwise dissimilarities δ_ij with weights w_ij, it defines a normalized weighted stress F(R) between the rarity-induced distances d_ij(R)=|R_i-R_j| and δ_ij. The main methodological contribution is DIT, a rarity meter obtained by minimizing F through a smoothed iterative algorithm for non-metric weighted unidimensional scaling, together with a scale-adjustment formula (Statement 1) and an out-of-sample kernel extension. Using the ROAR benchmark of 100 collections, the authors compare DIT with Rarity.tools, KRAMER, OpenRarity, NFTGo, and ROAR and report that DIT achieves the best F-based performance in over 65% of collections. The paper claims that DIT is 'optimal' in the NWUDS sense and that it 'demonstrates superior performance compared to existing methods.'","tokens_in":14917,"tokens_out":8917,"duration_ms":96770,"significance":"If the reported superiority could be established under an externally validated performance measure, the paper would make a useful contribution: it supplies a clean mathematical formulation, a correct scale-adjustment derivation in Statement 1, a reproducible open-source benchmark, and a computationally cheaper objective than the weighted correlation used by ROAR. The dimension-reduction link between rarity and trade dissimilarities is a reasonable research direction with potential applications beyond NFTs. At present, however, the headline empirical claim is not independently established: it rests on a metric that the proposed method itself optimizes, and no evidence is given that this metric ranks meters similarly to the established weighted correlation Fwc or predicts market outcomes.","major_comments":[{"comment":"The head-to-head comparison is circular with respect to the proposed objective. DIT is trained by minimizing F (Section 5, Algorithm 1), and Section 7.2 then adopts 'the NWUDS-inspired DIT's F as the performance measure, replacing the previously employed weighted correlation Fwc.' The competitors are only rescaled through Statement 1; they are not optimized for F. In these circumstances, DIT winning under F is expected and does not constitute independent evidence of market alignment. The manuscript provides no validation that F orders meters similarly to Fwc or any other external criterion. Please repeat the performance-profile analysis under Fwc, report the Fwc values for all six meters, and additionally report correlations between F and Fwc across collections; without this, the 'superior performance' claim in the abstract and Section 7.3 is not a falsifiable finding about NFT rarity.","section":"Section 7.2 and Figure 2"},{"comment":"The paper overstates its optimality guarantee. Theorem 1 establishes monotone decrease and convergence to a stationary point of the smoothed stress S_ε; Theorem 2 gives a condition under which the smoothed problem has a unique stationary point at zero; Theorem 3 gives an ε→0 approximation result. None of these implies that the final local minimization in Algorithm 1 reaches a global minimum of the original stress S. The sentence in Section 5 that the procedure 'ensures that the solution converges to a global minimum' and the Section 8 statement about 'convergence ... to a globally optimal solution' are therefore unsupported. Please either remove the global-optimality claims or provide a rigorous guarantee.","section":"Section 5, Theorems 1-3 and Section 8"}],"minor_comments":[{"comment":"In the KRAMER definition, the weights are denoted α_1,...,α_N, but the sum runs over T traits; this should be α_1,...,α_T.","section":"Section 3.2.2"},{"comment":"In the suggested ε-sequence, the formula for d* uses the index I in both the maximum and the sums; the collection size is N, so N should replace I.","section":"Section 5"},{"comment":"The name of the fourth comparator appears as 'NFTGO' in the text and 'NFTGo' in Figure 2; please harmonize the spelling.","section":"Section 7.3"},{"comment":"The text says the optimal k values ranged from 1 to 25, but Figure 3 displays only k=2,...,10; either expand the figure or clarify the range shown.","section":"Section 7.4"},{"comment":"The statement 'the optimal k was 2 in 60%' should specify that this is the fraction of collections, not of tokens or trades.","section":"Section 7.4"},{"comment":"The out-of-sample description would benefit from an explicit statement that the dissimilarity and weight matrices used in Algorithm 1 are computed from the training portion of the trades only, and that Figure 2's F values are computed on the test portion.","section":"Section 6.2 and Section 7.2"}],"recommendation":"major_revision","confidential_remarks":"The mathematical core of the paper, especially Statement 1 and the smoothing algorithm, is plausible and worth preserving, and the availability of code and data is a strength. The main empirical claim is currently undermined by the evaluation protocol: the benchmark metric is the same objective that DIT minimizes, and the established Fwc metric is never reported. I do not recommend rejection because the flaw appears fixable within the scope of a revision: evaluating all meters under Fwc and validating F against Fwc would directly address the circularity. The revision should be substantial, and the authors should also temper the global-optimality language unless a genuine guarantee is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: DIT is trained by minimizing the stress F, and then the paper compares all meters using that same F. In Section 7.2 they replace the established ROAR measure Fwc with F, so DIT is being graded on its home turf. The competitors are only given a scalar adjustment (Statement 1). Unsurprisingly, DIT wins. The stress-test note is right: the head-to-head is not independent evidence. Without a validation of F against Fwc or some external market outcome, the 'superior performance' claim is largely circular. This is the load-bearing flaw, and I don't see it fixed in the current text.\n\nThat said, the paper is not empty. It is the first to frame NFT rarity meter design as non-metric weighted unidimensional scaling, and it credits Pliner's machinery honestly. The DIT performance measure, though circularly used, is computationally more efficient than Fwc and could become a useful diagnostic if it were shown to rank meters similarly or to predict something. The theoretical section is mostly a competent adaptation of known smoothing and continuation arguments; Theorems 1–3 establish stationarity and a heuristic smoothing schedule, not global optimality, so the words 'optimal' and 'guaranteed' should be pulled back. I also notice Theorem 2's condition (ε > 4d*) implies the only minimum is the trivial zero configuration, which is a degenerate solution—if the algorithm is run there it would collapse all scores. That deserves a closer look, but it's not the main issue.\n\nReproducibility is genuinely helped by the public GitHub repository, and the dataset update to late 2024 is a plus. The preprocessing details are somewhat deferred, but the code is there.\n\nWho is this for? Researchers in blockchain analytics who want a new rarity metric and a faster benchmark measure. The paper deserves a serious referee, but only after the evaluation loop is broken: rerun the comparison under Fwc, validate F against Fwc, and soften the optimality claims. I'd send it back for major revision rather than desk-reject.","headline":"The paper's headline superiority claim is compromised by evaluating DIT under the very objective it minimizes, but the MDS framing and public code are real assets.","tokens_in":15467,"tokens_out":1064,"would_cite":false,"duration_ms":13681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A rarity meter fit to trade data beats five NFT rivals.","keywords":["NFT rarity meters","non-metric weighted unidimensional scaling","dimension reduction","Dissimilarity in Trades (DIT)","ROAR benchmark","weighted stress","blockchain"],"falsifier":"Rerun the ROAR comparison with the established weighted-correlation measure $F_{wc}$ on the same held-out trades and scale-adjusted meters; if DIT no longer wins on most collections, its reported edge comes from optimizing the benchmark rather than from market alignment. A second check is to take labeled wash-traded or manipulated NFTs and see whether DIT's most extreme scores flag them more accurately than the interpretable meters.","tokens_in":14458,"feed_emoji":"🖼️","tokens_out":9883,"duration_ms":98742,"temperature":0.7,"pith_summary":"This paper claims that NFT rarity is best modeled as a one-dimensional summary of pairwise trade dissimilarities, and that this point of view yields both a better meter and a cheaper way to evaluate meters. It introduces Dissimilarity in Trades (DIT), a rarity score obtained by minimizing the non-metric weighted unidimensional scaling stress $F$ between pairwise rarity distances and dissimilarities built from close-in-time NFT deals. On the ROAR benchmark of 100 collections, DIT achieves the best $F$ on over 65% of collections, ahead of Rarity.tools, KRAMER, OpenRarity, NFTGo, and ROAR. DIT's scores are non-interpretable: they are optimized numbers rather than transparent trait formulas. The paper's new performance measure $F$ works on $N\\times N$ matrices after one aggregation of trade pairs, which it argues makes large-scale evaluation much faster than the earlier weighted-correlation measure.","feed_headline":"A rarity meter fit to trade data beats five NFT rivals","feed_subtitle":"DIT wins on over 65% of 100 collections by aligning rarity gaps with real trade dissimilarities.","key_machinery":"The load-bearing object is the NWUDS stress $F$ with its dissimilarity matrix $\\delta_{ij}$ and weight matrix $w_{ij}$, where each weight aggregates a time-kernel across close-in-time deals of the same token pair and each dissimilarity is the weighted mean absolute log-price gap for that pair. The optimization follows Pliner's smoothing construction: $S_\\varepsilon(\\mathbf{x})$ is the raw stress averaged over an $\\varepsilon$-cube, and the minimum is approached by the fixed-point iteration $x^{q+1}_i = \\sum_j w_{ij}(x^q_j - d_{ij}u_\\varepsilon(x^q_i-x^q_j))/\\sum_j w_{ij}$, with $\\varepsilon$ annealed from $2d^*$ toward zero and a final local minimization of the raw stress. The proof of convergence identifies the iteration with gradient descent at step size $1/(2N)$ and invokes Polyak's theorem; Theorem 2 gives a regime ($\\varepsilon > 4d^*$) where the smoothed stress has a unique minimum. Out-of-sample tokens receive DIT scores by kernel regression on an interpretable rarity meter $\\tilde{R}$, with the neighbor count $k$ chosen by cross-validation.","core_discovery":"The paper's central claim is that the rarity-meter design problem reduces to non-metric weighted unidimensional scaling (NWUDS): from trade data one builds weighted dissimilarities $\\delta_{ij}$ between token pairs, and an optimal rarity vector $\\mathbf{R}$ minimizes the normalized stress $$F(\\mathbf{R}) = \\sqrt{ \\frac{\\sum_{i,j} w_{ij}(|R_i-R_j|-\\delta_{ij})^2}{\\sum_{i,j} w_{ij}|R_i-R_j|^2} }.$$ It argues that this objective is the right market-alignment criterion, because rarity gaps should reproduce price gaps between deals that happen close in time, and that smoothing the stress over a cube of side $\\varepsilon$ turns the multiextremal problem into one that annealing can solve. The resulting DIT meter is non-interpretable by design, yet on the test portion of the ROAR benchmark it records the best $F$ on more than 65% of the 100 collections, with ROAR and NFTGo following. A supporting contribution is the claim that $F$ itself is a better practical performance measure than the established weighted correlation $F_{wc}$, because it is cheaper to compute.","pith_inferences":["The benchmark is tilted toward DIT: all competitors are scored with $F$, the same objective DIT was optimized to minimize, so the claimed margin may shrink or vanish under the earlier weighted-correlation measure $F_{wc}$ or an independently defined price target.","Because $k=2$ is the optimal neighbor count in roughly 60% of collections, a fixed default of two neighbors might be nearly as good as per-collection selection, but the paper does not report that comparison.","The paper's wash-trading and market-manipulation applications are forward-looking rather than tested; a direct test would compare DIT's outlier scores against labeled instances of known manipulation.","The dimension-reduction recipe should transfer to non-blockchain collectibles or any asset with transaction histories, though the paper only demonstrates it on NFTs."],"forward_implications":["If $F$ is accepted as the criterion, DIT's scores track the pairwise price structure of NFT trades more closely than any of the five interpretable meters tested.","Because $F$ needs only the aggregated $N\\times N$ dissimilarity and weight matrices, evaluating a rarity meter on large collections is much cheaper than recomputing over pairwise deals, which makes benchmarking on big collections practical.","DIT's loss of interpretability is not fatal for automation: the scores can be fed into ranking, pricing, and anomaly-detection tools, and the paper argues they are suitable for spotting wash trading and pump-and-dump patterns.","The same construction carries over to any one-dimensional score meant to mirror pairwise dissimilarities, so the framework is reusable beyond NFT collections."],"supporting_citations":[{"why":"Defines the Rarity.tools inverse-frequency formula used as one of the baseline meters.","marker":"[7]"},{"why":"Introduces the KRAMER interpretable meter and the weighted-correlation performance measure that DIT generalizes and competes against.","marker":"[8]"},{"why":"Provides the ROAR benchmark dataset, the train/test split, and the ROAR ensemble meter used in the evaluation.","marker":"[9]"},{"why":"Supplies the smoothed-stress formulation and iterative minimization idea that the DIT algorithm extends to arbitrary weights.","marker":"[41]"},{"why":"Addresses the multiextremal stress and global-optimization problem that motivates the annealing schedule.","marker":"[42]"},{"why":"Provides the gradient-convergence theorem used to prove the fixed-point iteration reaches a stationary point.","marker":"[47]"},{"why":"Underlies the nonparametric kernel regression used for out-of-sample extension of DIT scores to new tokens.","marker":"[50]"},{"why":"Hosts the released data and scripts that make the ROAR comparisons reproducible.","marker":"[51]"}],"fun_headline_variants":["Dimension reduction crafts NFT rarity meter that beats rivals","Trade-data rarity meter wins on most NFT collections","DIT: New NFT rarity score tops 65% of benchmark collections","Optimal NFT rarity via non-metric scaling beats existing meters","Rarity meter from dimension reduction outperforms on ROAR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gap between two tokens' rarity scores ought to reproduce the gap between their close-in-time trade prices; the paper then scores every competitor with that same yardstick, which is also the yardstick its own meter was built to minimize.","fun_headline_variants_meta":{"raw":{"variants":["Dimension reduction crafts NFT rarity meter that beats rivals","Trade-data rarity meter wins on most NFT collections","DIT: New NFT rarity score tops 65% of benchmark collections","Optimal NFT rarity via non-metric scaling beats existing meters","Rarity meter from dimension reduction outperforms on ROAR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3036,"prompt_tokens":984,"completion_tokens":2052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1981}},"tokens_in":600,"tokens_out":2052,"duration_ms":15180,"temperature":1.0,"reasoning_tokens":1981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:19:41.702778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the ROAR comparison with the established weighted-correlation measure $F_{wc}$ on the same held-out trades and scale-adjusted meters; if DIT no longer wins on most collections, its reported edge comes from optimizing the benchmark rather than from market alignment. A second check is to take labeled wash-traded or manipulated NFTs and see whether DIT's most extreme scores flag them more accurately than the interpretable meters.","supporting_citations":[{"cited_title":"URL https://raritytools.medium.com/ranking-rarity- understanding-rarity-calculation-methods-86ceaeb9b98c","cited_arxiv_id":null,"evidence_quote":"Defines the Rarity.tools inverse-frequency formula used as one of the baseline meters."},{"cited_title":"Belousov, M","cited_arxiv_id":null,"evidence_quote":"Provides the ROAR benchmark dataset, the train/test split, and the ROAR ensemble meter used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the smoothed-stress formulation and iterative minimization idea that the DIT algorithm extends to arbitrary weights."},{"cited_title":"Pliner, Metric unidimensional scaling and global optimiza- tion, J","cited_arxiv_id":null,"evidence_quote":"Addresses the multiextremal stress and global-optimization problem that motivates the annealing schedule."},{"cited_title":"Polyak, Introduction to Optimization, Nauka, 1983","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-convergence theorem used to prove the fixed-point iteration reaches a stationary point."},{"cited_title":"Yanovich, Asymptotic Properties of Nonparametric Estima- tion on Manifold, JMLR Workshop and Conference Proceedings 60 (2017) 18–38","cited_arxiv_id":null,"evidence_quote":"Underlies the nonparametric kernel regression used for out-of-sample extension of DIT scores to new tokens."},{"cited_title":"URL https://github.com/fasghq/DIT-benchmark/tree/main","cited_arxiv_id":null,"evidence_quote":"Hosts the released data and scripts that make the ROAR comparisons reproducible."}],"review_version":2}