{"id":"8be8916d-b862-4f05-831f-30c55b233de7","arxiv_id":"2411.16298","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A replication study finds that Rank-N-Contrast improves regression loss on AgeDB and withstands a missing-age-range holdout better than plain L1, but shows less benefit on a small mango dataset.","lead":"This paper reproduces the Rank-N-Contrast loss for image regression and adds a holdout test that removes an age range from training. It reports that RNC lowers regression loss on a large face dataset but gives less benefit on a small mango dataset.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests only on training/validation loss curves; no task metric (MAE/RMSE), no seeds, and no hyperparameter report is given, so the reported 'better performance and robustness' is not actually measured.","rationale":"I read the paper as a replication/extension study whose central claim is empirical: adding L_RNC at the encoder improves regression accuracy and robustness compared with plain L1. The weakest point is not internal consistency of the RNC math—that is inherited from the original paper—but the evidential link between L_RNC and measured regression performance. The manuscript reports only loss curves and a final auxiliary RNC loss value, not the actual regression error on held-out data. Loss curves are not task metrics, especially when the two compared pipelines are trained differently: the RNC model receives an additional encoder training phase, so lower optimizer loss does not imply lower test error. The robustness experiment is also reported only through curve shapes and a single final loss number, leaving the 'more robust' claim unquantified. The mango result is an important qualification that is mentioned but not analyzed; it suggests the benefit may depend on dataset size, which the conclusion partially acknowledges with 'with enough data.' The reader's weakest assumption about pipeline identity is on target, and I extend it: even the outcome variable (task error) is never reported. No fatal flaw is present, but the central empirical conclusion is unverified as written. A conditional revision that adds a metrics table, multiple seeds, and explicit qualification of the mango result would be appropriate, matching the reader's CONDITIONAL verdict.","tokens_in":5428,"tokens_out":2416,"duration_ms":24370,"concrete_test":"Report a table of held-out MAE (and RMSE if desired) for L1 and RNC+L1 on AgeDB normal split, AgeDB with ages 30-40 removed from training, and MangoMassNet-552, using identical encoder, optimizer, and hyperparameters across at least 3 random seeds, with mean +/- std. If RNC+L1 held-out error is not lower than L1 by a margin exceeding seed variance on both AgeDB splits, the paper's 'better performance and robustness' claim fails. For the mango dataset, if RNC+L1 is not better, the paper must explicitly qualify the claim to datasets of sufficient size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's conclusion ('using L_RNC at the encoder can help achieve better performance for regression tasks and make the model more robust to missing data') is an empirical claim about regression accuracy, but the only quantitative evidence reported is the final RNC loss at the encoder (Table I) plus qualitative inspection of the training/validation loss curves (Figs. 1-6). L_RNC is an auxiliary representation loss; a lower value of that loss (e.g., 3.92 vs. whatever the L1 model achieves) does not by itself establish lower MAE or RMSE for the regression task. Moreover, the L1 and RNC+L1 pipelines are not shown to be identical: Section VI says the RNC model first trains the encoder with L_RNC and then trains the whole model with L1, while the L1 model is trained with L1 throughout, and no learning rates, batch sizes, epochs, optimizer settings, seeds, or compute budgets are reported. The mango experiment is a potential counterexample within the paper: the L1 model converges faster than RNC+L1 on both training and validation loss, which the authors hand-wave as 'the learner at the encoder is not learning as much' without reporting final prediction error. Thus the central claim may be true, but this manuscript has not demonstrated it; the gap is a missing measurement, not a theoretical inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a replication/extension study of the Rank-N-Contrast (RNC) method for regression. The authors compare a model trained with standard L1 loss against a model that first trains the encoder with the RNC loss and then fine-tunes with L1, across three settings: the AgeDB dataset, an AgeDB variant with ages 30-40 removed from training, and the smaller MangoMassNet-552 dataset. The paper concludes that RNC yields better regression performance and robustness to missing data. The evidence presented is limited to training/validation loss curves (Figures 1-6) and the final RNC loss at the encoder (Table I); no held-out task metric such as MAE or RMSE is reported.","tokens_in":5790,"tokens_out":4682,"duration_ms":40810,"significance":"If substantiated, the manuscript would provide a useful independent check on the original RNC claim and extend it to a new small dataset and a missing-range generalization test. Strengths include the choice of an additional real-world regression dataset (mango mass estimation), a deliberately challenging holdout split, and transparency about reusing the original implementation (Section IV-A). However, the current evidence does not establish the claimed benefit: no regression accuracy metric is reported, no seeds or hyperparameters are given, and the mango experiment visually contradicts the benefit claim. The contribution is therefore presently a description of an experimental setup plus qualitative loss-curve observations, rather than a validated replication result.","major_comments":[{"comment":"The central claim that RNC improves regression performance and robustness is not measured by any task metric. The only quantitative result reported is the final RNC loss at the encoder (Table I), and the supporting evidence is qualitative inspection of training/validation loss curves (Figures 1-6). A lower auxiliary representation loss or a lower L1 validation loss during training does not establish lower MAE or RMSE on a held-out test set, particularly because the prediction heads and training schedules differ between the two pipelines. Please report final test-set MAE/RMSE (with standard deviations over multiple seeds) for L1 and RNC+L1 on AgeDB, modified AgeDB, and MangoMassNet.","section":"Section VI / Table I"},{"comment":"The manuscript does not report any hyperparameters, optimizer settings, learning rates, batch sizes, epoch counts, seed values, or compute budgets, so the reader cannot verify that the L1 and RNC+L1 pipelines are otherwise identical. The two pipelines also differ by design (RNC+L1 first trains the encoder with L_RNC, then trains the whole model with L1, whereas L1 is trained end-to-end), so attribution of performance differences to L_RNC requires demonstrating that the comparison is controlled. Please include a complete experimental setup table and, ideally, release the code and seeds.","section":"Section IV-A and Section VI"},{"comment":"The mango dataset experiment is a counterexample to the paper's benefit claim: the L1 model converges faster than RNC+L1 on both training and validation loss. The explanation offered (\"the learner at the encoder is not learning as much\") is speculative, and no final prediction error is reported for either model. This result, as presented, undermines the general claim that RNC helps regression tasks and must be addressed with a task-metric comparison on this dataset rather than a post-hoc verbal explanation.","section":"Section VI, Figures 5-6"},{"comment":"The abstract claims the study \"achieve[s] state-of-the-art performance,\" but no comparison is made to any state-of-the-art method beyond the L1 baseline; SupCon, DINO, and SimCLR are discussed in Section II but never evaluated. Please either remove the state-of-the-art claim or add the corresponding baselines.","section":"Abstract and Section IV"}],"minor_comments":[{"comment":"The sentence \"We obtain two trained regression models that we can for which we can compare the performances\" is ungrammatical and should be rewritten.","section":"Section VI, AgeDB paragraph"},{"comment":"The phrase \"which significantly less than the previous dataset\" is missing a verb; it should be \"which is significantly less than...\".","section":"Section VI, MangoMassNet paragraph"},{"comment":"All figures should include axis labels, legends, and captions specifying exactly which loss is plotted (e.g., L1 prediction loss or total loss), so that the curves are interpretable without relying on the prose.","section":"Figures 1-6"},{"comment":"Table I reports only the final RNC loss at the encoder; including the final validation L1 loss for both L1 and RNC+L1 models would make the comparison more informative and would partially address the missing quantification in the text.","section":"Table I"},{"comment":"This section lists potential challenges but the paper never returns to them in the results or discussion; either connect these items to the experimental findings or remove the section to avoid a dangling outline.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a course-project report and currently does not meet the evidentiary standard of a research paper. The replication is not independent because the original implementation is reused, and the added empirical content is thin. If the authors add task metrics with multiple seeds, full hyperparameters, and a controlled comparison, the manuscript could become a useful reproducibility study. Given the current gap, I recommend major revision and would only consider acceptance after substantial experimental additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a student replication of Rank-N-Contrast (RNC) with a small robustness twist, and the write-up is honest about being a replication. But the central claim — that RNC improves regression performance and robustness — is never actually measured. What you get instead are training and validation loss curves for the RNC loss itself, plus a table of final RNC loss values. Lower RNC loss at the encoder does not equal lower MAE or RMSE on the regression task, and the abstract's \"state-of-the-art performance\" is asserted without any SOTA comparison.\n\nWhat's genuinely there: the authors clearly describe their setup, reuse the original RNC code as they say, and add two reasonable experiments: a holdout of ages 30–40 on AgeDB and a smaller mango-weight dataset. The idea of testing robustness to missing target ranges is sensible, and the AgeDB loss curves do show RNC converging faster and to a lower loss than plain L1. That part is a fair, if shallow, reproduction.\n\nThe soft spots are proportionate to how soft they actually are. No final MAE/RMSE anywhere, so the main empirical benefit is inferred, not shown. No seeds, no error bars, no hyperparameter reporting (learning rate, batch size, epochs, optimizer), so you cannot rule out confounding between the RNC loss and other training choices. The mango experiment—where L1 converges faster than RNC+L1 on both train and validation—is a direct counterexample to the generality claim, and the authors hand-wave it with \"the learner at the encoder is not learning as much,\" without quantifying prediction error. That is a gap in the evidence, not a theoretical contradiction, but it is a load-bearing gap.\n\nWho gets value: this is useful as a cautionary example for a reading group on replication standards, or as a starting point for someone who wants to replicate RNC and needs a minimal baseline. It is not a contribution that changes anyone's assessment of RNC.\n\nMy recommendation for peer review: I would not send this to referees in its current form. The claim is measurable and the authors just haven't measured it. A resubmission with a proper metrics table (MAE/RMSE on both datasets), multiple seeds, and a toned-down abstract would be worth another look, but as it stands the manuscript doesn't support its own conclusion.","headline":"A well-intentioned replication of Rank-N-Contrast that never actually measures the thing it claims (regression accuracy), so the central conclusion is unsupported as written.","tokens_in":6207,"tokens_out":1576,"would_cite":false,"duration_ms":16649,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This replication claims that adding the Rank-N-Contrast loss at the encoder improves regression accuracy and robustness to missing target ranges, provided the dataset is large enough.","keywords":["Rank-N-Contrast","regression","contrastive learning","representation learning","loss function","age estimation","robustness","missing data"],"falsifier":"Run the same AgeDB split with identical hyperparameters, optimizer, and seeds for L1 and RNC+L1; if the RNC model's validation loss on the full age range, including ages 30--40, is not below the L1 model's, or if the gap reverses under re-seeding, the paper's central robustness claim fails.","tokens_in":5189,"feed_emoji":"📉","tokens_out":10418,"duration_ms":82792,"temperature":0.7,"pith_summary":"This replication study asks whether the Rank-N-Contrast loss, which orders an encoder's embeddings by the rank of the target value, benefits regression models in practice. The authors compare a plain L1 pipeline with a pipeline whose encoder is first trained with $L_{RNC}$ and then combined with L1 prediction, using the AgeDB face-age dataset. They report that the RNC-trained model converges faster and reaches roughly half the L1-only training loss, and that it degrades less when ages 30--40 are removed from the training set. Extending the test to a 552-image mango-weight dataset, the advantage reverses: L1 converges faster and the final $L_{RNC}$ value is higher. The paper concludes that, given enough data, using $L_{RNC}$ at the encoder improves regression performance and robustness to missing ranges in the training set.","feed_headline":"Rank-N-Contrast loss cuts regression loss and survives missing ages","feed_subtitle":"Replication on AgeDB shows it halves loss and copes with missing age ranges; the edge fades on a 552-image mango dataset.","key_machinery":"The load-bearing object is the Rank-N-Contrast loss $L_{RNC} = \\frac{1}{2N}\\sum_{i=1}^{2N} \\ell_{RNC}(i)$, where each per-anchor term contrasts the anchor's embedding with other samples in the batch weighted by their rank in label space: samples with closer target values contribute larger similarity terms in a temperature-scaled softmax. This loss is what forces the encoder to place samples with nearby continuous labels nearby in the embedding, giving the later L1 predictor a representation that respects label order. The paper's experiments vary only this loss at the encoder while keeping the same predictor loss.","core_discovery":"The claim the authors are trying to establish is that a regression-aware contrastive loss at the encoder produces a latent space ordered by the target value, and that this ordering carries over into better and more robust predictions. On AgeDB, $L_{RNC}+L1$ reaches a final training loss near 1 while L1 converges near 2, and its validation curve starts low and stays smoother. With ages 30--40 missing from training, both models' validation losses rise from 6.75 to 8.5, but the L1 model's convergence becomes notably noisier and slower, which the authors read as evidence that the continuous embedding helps extrapolate into the missing range. On MangoMassNet-552, the pattern breaks: L1 converges faster on both training and validation, and the final encoder $L_{RNC}$ loss is 5.37 versus 3.92--3.98 on AgeDB. The stated conclusion is therefore conditional: with enough data, $L_{RNC}$ at the encoder helps regression and robustness.","pith_inferences":["The paper leaves implicit that batch composition controls when RNC helps: if a batch contains few distinct target values, the ranking signal weakens, which may explain why the 552-image mango dataset loses the advantage.","The missing-age-range result suggests a potential application to regression settings with censored or sparse labels, such as medical or pricing data, but the paper tests only age on one dataset, so transfer remains an open question.","A natural next experiment is to vary AgeDB size continuously, for example from 500 to 16,000 images, to map where the RNC advantage over L1 appears or disappears."],"forward_implications":["With enough training data, adding $L_{RNC}$ to the encoder lowers final regression loss and speeds convergence relative to L1-only training.","The continuous ordering learned by $L_{RNC}$ makes validation performance degrade more gracefully when a contiguous range of target values is absent from training.","The benefit is not automatic on small datasets: on 552 images, L1 converges faster, so RNC's usefulness appears to scale with data or label diversity.","RNC is a plug-in loss for the encoder, so it can be attached to existing regression predictors without changing the prediction layer."],"supporting_citations":[{"why":"It supplies the Rank-N-Contrast loss, its theory, and the original benchmark results this paper is replicating.","marker":"[8]"},{"why":"It supplies the AgeDB face-age dataset used for the main L1 versus RNC+L1 comparison and for the missing-age-range split.","marker":"[5]"},{"why":"It supplies the MangoMassNet-552 image dataset used to test whether the benefit persists on a smaller dataset.","marker":"[3]"},{"why":"It provides supervised contrastive learning, the representation-learning baseline that motivates the continuity argument because it treats bins as classes.","marker":"[4]"}],"fun_headline_variants":["Rank-N-Contrast helps age regression but fades on mango data","RNC loss halves age error, but edge disappears on small dataset","Missing ages? RNC adapts, but L1 wins on mango dataset","Continuous contrastive loss robust to missing data, not universal","Rank-N-Contrast: robust on faces, weaker on mango regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the L1 baseline and the RNC+L1 model are trained in exactly the same way except for the added representation loss, but it gives no hyperparameters, seeds, or training budget to back that up.","fun_headline_variants_meta":{"raw":{"variants":["Rank-N-Contrast helps age regression but fades on mango data","RNC loss halves age error, but edge disappears on small dataset","Missing ages? RNC adapts, but L1 wins on mango dataset","Continuous contrastive loss robust to missing data, not universal","Rank-N-Contrast: robust on faces, weaker on mango regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1491,"prompt_tokens":923,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":475}},"tokens_in":539,"tokens_out":568,"duration_ms":6337,"temperature":1.0,"reasoning_tokens":475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:15:37.946387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same AgeDB split with identical hyperparameters, optimizer, and seeds for L1 and RNC+L1; if the RNC model's validation loss on the full age range, including ages 30--40, is not below the L1 model's, or if the gap reverses under re-seeding, the paper's central robustness claim fails.","supporting_citations":[{"cited_title":"Dex: Deep expectation of apparent age from a single image","cited_arxiv_id":null,"evidence_quote":"It supplies the Rank-N-Contrast loss, its theory, and the original benchmark results this paper is replicating."},{"cited_title":"Estimating mango mass from rgb image with convolutional neural network","cited_arxiv_id":null,"evidence_quote":"It provides supervised contrastive learning, the representation-learning baseline that motivates the continuity argument because it treats bins as classes."}],"review_version":1}