Instance Hardness-Based Relevance for Imbalanced Regression

George D. C. Cavalcanti; Juscimara G. Avelino; Rafael M. O. Cruz; Vitor M. Leitao

REVIEW 4 major objections 4 minor 20 references

Instance Hardness-Based Relevance for Imbalanced Regression

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Imbalanced regression should treat hard-to-predict instances as rare, not only infrequent target values.

desk verdict InHaR is a genuinely new twist on relevance for imbalanced regression, but the empirical comparisons may be compromised by an unstated fold-handling protocol, and the abstract overstates the results. read the letter →

arxiv 2607.20173 v2 pith:RJZLUE76 submitted 2026-07-22 cs.LG

Vitor M. Leitao , Juscimara G. Avelino , George D. C. Cavalcanti , Rafael M. O. Cruz This is my paper · ORCID

classification cs.LG

keywords imbalancedregressioninstancehardnessrelevancefunctionresamplingrandomoversamplingGaussiannoisebimodaldistributionrareidentification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that for imbalanced regression, rarity should be measured by how hard an instance is for a learning algorithm, not by how infrequent its target value is. It proposes InHaR, a relevance function that scores each instance by the normalized prediction error of a pool of regressors, and uses a fixed threshold to label a subset as rare. Unlike the standard target-frequency relevance function, InHaR can flag hard instances inside dense regions, such as the valley between two modes of a bimodal target. When InHaR guides random oversampling and Gaussian-noise resampling, the paper reports improved MAE and MSE on 29 datasets plus a synthetic bimodal study, with most pairwise wins statistically significant. The point of caring: resampling decisions currently built on a density-only notion of rarity may be targeting the wrong instances.

What carries the argument

The load-bearing object is the Instance Hardness-based relevance function (InHaR), which replaces the target-value-to-relevance mapping phi(y) with a per-instance difficulty score: IH = 1 - average over a pool of regressors of exp(-relative squared error / gamma). The exponential-error form, normalized by gamma = mean(y^2), turns any regressor's prediction error into a [0,1] hardness; a fixed threshold tau = 0.7 then partitions the dataset into rare and normal sets for resampling. The identity does the work: it lets the attribute space and learning behavior enter the rarity decision, which density-only relevance cannot.

What would settle it

Run the bimodal experiment with a known generative mixture (e.g., two Gaussians with known weights) and define true rare regions by the mixture's low-density components; if InHaR's top-30% median-hardness bins do not align with those regions, or if density-based resampling beats it on MAE/MSE there, the identification claim fails. Simpler: compute IH with a regressor pool disjoint from the downstream model and see whether the resampling wins disappear.

Watch

Extended reading notes

Core claim

The central claim is that learning-based instance hardness is a more useful notion of rarity for imbalanced regression than target marginal density. Traditional relevance functions assign fixed importance to extreme or low-frequency target values and fail in bimodal distributions, where the low-density region between modes is not genuinely rare. InHaR defines relevance directly as IH_L(x_i, y_i) = 1 - (1/|L|) sum_j exp(-d(y_i, h_j(x_i)) / gamma), with gamma the mean squared target, so instances that are consistently hard to predict get high relevance regardless of their frequency. The paper shows that IH and the traditional relevance function are only partially correlated, that InHaR flags r

Load-bearing premise

The method assumes that an instance's prediction difficulty, measured on the same kinds of regressors that will be trained later, reveals which instances are worth duplicating during resampling—rather than reflecting noise or outliers that amplification would make worse.

Editorial extensions

If this is right

Relevance-based resampling need not be tied to target marginal density; any hardness measure can drive RO and GN while preserving the existing workflow.
In bimodal target distributions, InHaR can mark mid-distribution instances as rare, so resampling can focus on the hard valley rather than only the tails.
Because IH and the traditional relevance function are only partially correlated, the two definitions select different instances, so the choice of relevance criterion changes which regions a model ends up improving.
Using relevance-free metrics for comparison shows that hardness-guided resampling can improve normal-region performance rather than degrading it, unlike aggressive oversampling of statistical extremes.
The method generalizes across five regressor families, suggesting the hardness signal is not tied to one learning algorithm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable next step would be to calibrate the threshold tau per dataset or to replace the fixed 0.7 with a data-driven quantile; if the gains shrink under cross-validated selection, the fixed threshold is doing more work than the hardness principle.
The same hardness-based relevance could be plugged into synthetic oversampling variants beyond RO and GN; if IH identifies hard regions, synthetic oversampling should benefit at least as much as duplication.
A stronger validation of the bimodal claim would define true rarity from the known data-generating mixture rather than from IH itself; absent that, the 'correctly identifies' result is partly self-referential.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper has a new idea—using instance hardness as the relevance criterion for imbalanced regression—and it mostly runs a sensible evaluation, but the central empirical claim is currently unverifiable because the manuscript never says how IH is computed relative to the cross-validation folds. If IH is computed on the full dataset, test target values leak into the choice of which instances get oversampled and the comparisons in Tables III-V don't mean what they claim. Even if it's fold-local, using the same model families that are later evaluated is a milder form of selection bias. The code link may resolve this, but the paper as written does not.

What's genuinely good: replacing statistical rarity with learning difficulty is more than a parameter tweak. The correlation analysis between IH and the Ribeiro relevance function is interesting; it shows they are related but not identical, which motivates the approach. Using MAE/MSE rather than SERA or similar is the right call for a paper that is changing the relevance definition. The Wilcoxon testing is appropriate, and the 29-dataset comparison is a reasonable amount of work.

The soft spots, in decreasing order: the fold-protocol ambiguity above is the big one. Then the abstract's 'approximately 70%' is contradicted by the paper's own Table III, which gives 68/145 (47%) for MAE and 58/145 (40%) for MSE. That matters because the abstract is what people will remember. The bimodal validation is circular: the paper defines rare bins as those with high median IH, then reports that those bins have high IH. Without an external ground-truth definition of rarity, 'correctly identifies' is meaningless. Finally, the threshold τ=0.7 and the 70th percentile are chosen after preliminary experiments, with no sensitivity analysis; the paper honestly acknowledges this in the conclusion, but it remains a major limitation.

This is a paper for the imbalanced-regression subcommunity. The idea is worth a serious referee, and the authors have a chance to fix things. But in its current form I would not cite it for the main empirical claim. My recommendation: send it to peer review with a request that the reviewers verify the IH computation protocol and ask for a sensitivity analysis and a corrected abstract.

Referee Report

4 major / 4 minor

Summary. The paper proposes InHaR, an Instance Hardness-based relevance function for imbalanced regression, replacing the traditional target-frequency relevance function ϕ by a difficulty-based score computed from prediction errors of a set of regressors (Eq. 1, Section III). Rare instances are then selected by thresholding IH values, and these instances guide Random Oversampling (RO) and Gaussian Noise (GN) resampling. Experiments use 2×5 cross-validation on 29 datasets and a synthetic bimodal dataset, comparing MAE and MSE against no resampling, the traditional relevance function, and the Distance-Based Relevance Function (DRF). The authors claim that InHaR correctly identifies rare regions under bimodal distributions and improves predictive performance in approximately 70% of the tests compared to traditional relevance-based approaches.

Significance. If the empirical claims hold, the paper offers a novel and potentially useful perspective on relevance in imbalanced regression by shifting from distributional rarity to learning difficulty. The use of relevance-independent metrics (MAE and MSE), the Wilcoxon signed-rank tests for paired comparisons, and the public release of code and data are positive features. However, the currently unverifiable protocol for computing IH, the circular validation in the bimodal experiment, and the inconsistent reporting of the headline '70%' result prevent the paper from being accepted in its present form. The central idea is promising, but the evidence needs to be substantially clarified and strengthened.

major comments (4)

[Section IV - Instance Hardness / Evaluation Methods] The manuscript never states whether IH values in Eq. (1) are computed within each training fold (e.g., by internal cross-validation) or on the full dataset before resampling. Section IV says IH was 'instantiated using error-based measures computed from the regression models defined in the Regression Models subsection' — the same model families (RF, BG, XGB, SVR, MLP) used later as base learners in the 2×5 CV. If IH is computed on all instances, test target values leak into the choice of which instances are oversampled, invalidating the comparisons in Tables III–V. Even if computed fold-locally, the overlap between IH regressors and evaluated regressors should be described and justified. This is a load-bearing methodological ambiguity and makes the central empirical claim currently unverifiable from the text alone.
[Section V-B - Bimodal Distributions] The validation in the bimodal experiment is circular. Bins are labeled 'rare' when their median IH is in the top 30% (the 70th percentile), and the paper then claims that InHaR 'correctly identifies' rare regions (Abstract and Section V-B). No independent ground truth for rarity is provided; the method is shown to identify regions that it itself defines as rare. Furthermore, the 70th percentile was 'selected based on preliminary experiments' with no sensitivity analysis, as the paper itself acknowledges. This section should either provide an external benchmark (e.g., known under-sampled areas in the synthetic data) or be reframed as a descriptive demonstration rather than a validation.
[Abstract / Introduction and Table III] The abstract states that 'in approximately 70% of the tests performed, the proposed approach achieved superior performance compared to the traditional approach.' This is not supported by Table III, where InHaR-RO and InHaR-GN together win 68/145 (47%) MAE pairs and 58/145 (40%) MSE pairs. Even if 'traditional approach' is intended to mean only the classical RO/GN (Table IV), the MAE win rate is 51/58 (88%) and the MSE rate is 40/58 (69%), not an overall 70%. The claim needs a precise definition of 'tests' and a consistent number.
[Section IV / Conclusion - Threshold sensitivity] The method relies on a fixed threshold τ=0.7 for IH (Section IV) and a 70th-percentile cutoff in the bimodal analysis (Section V-B), both chosen based on preliminary experiments. The conclusion explicitly notes that 'a systematic analysis of threshold sensitivity was not conducted.' Since the performance of InHaR is directly controlled by these hyperparameters, the reported results may not generalize. The authors should provide a sensitivity analysis or otherwise justify that the performance advantage is not an artifact of the chosen thresholds.

minor comments (4)

[Section V-B] There is a typo: 'future work. .' should be 'future work.'
[Table III] The row/column formatting makes the comparison hard to read. Consider providing a clearer table with pairwise wins/losses or a visual matrix to avoid misreading.
[Section V-A] Figure 2 would benefit from a caption that explicitly states over how many datasets the correlation distribution is plotted and how the IH values were computed for this analysis (e.g., full-data vs. cross-validated predictions).
[References] Reference [18] is a technical report; if a peer-reviewed version of the instance-hardness-for-regression measure exists, it should be cited instead.

Circularity Check

1 steps flagged · score 4.0 of 10

Bimodal rare-region validation is definitional, but the main resampling comparison is independently grounded.

self definitional [Section V-B (Applying InHaR to Bimodal Distributions), Figure 3 and surrounding text]
"This identification is performed by discretizing the target variable into equidistant bins and computing the median Instance Hardness value within each bin. A global threshold is then defined, and bins whose median IH falls within the top 30% of observed values—corresponding to the 70th percentile of the IH distribution in our experiments—are labeled as rare."

InHaR defines rarity as high Instance Hardness (Algorithm 1, Eq. 1). In the bimodal validation, 'rare regions' are defined by exactly the same criterion: bins whose median IH is in the top 30% are labeled rare. The subsequent claim that InHaR 'correctly identifies rare regions' therefore reduces to showing that high-IH bins have high IH—true by construction. No independent ground truth for rarity (e.g., a known under-sampled generating region) is supplied, so this supporting validation is self-referential.

full rationale

The core empirical comparison (Tables III–IV) is not circular: the proposal is that replacing a target-frequency relevance function with an error-based hardness score in RO/GN should improve MAE/MSE, and these metrics do not encode the relevance criterion. Nothing in Eq. (1) mathematically forces InHaR-guided resampling to win on those metrics across datasets; the observed wins are contingent empirical outcomes. The self-citation to Avelino et al. [6] is used only to justify choosing RO and GN and is not load-bearing. The fixed τ=0.7 is a design parameter, not a fitted prediction. However, the bimodal validation in Section V-B is self-referential: 'rare regions' are operationalized as bins with median IH in the top 30%, and Figure 3 is then presented as evidence that InHaR correctly identifies rare regions. Without an independent rarity ground truth, this reduces to restating the method's own thresholding rule. A separate leakage risk exists because the manuscript does not state whether IH is computed fold-locally or on the full dataset; if the latter, test labels would influence the oversampling selection, but the text is not explicit enough to count this as a demonstrated circular step. Overall, the central resampling claim is independently grounded, but one supporting validation is definitional, giving score 4.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical assumption that hardness correlates with useful oversampling targets, plus a circular validation in the bimodal case. The free parameters are the threshold and percentile, both tuned post hoc.

free parameters (2)

Relevance threshold tau = 0.7
Fixed threshold used to split IH into rare/normal; adopted based on preliminary experiments, no sensitivity analysis (Section IV, V-C).
Bimodal rare-bin percentile = 70th percentile (top 30%)
Bins whose median IH is in top 30% labeled rare; selected based on preliminary experiments (Section V-B).

assumptions (4)

domain assumption Instance hardness IH_L(x_i,y_i) as defined by Eq. 1 accurately quantifies the learning difficulty of each instance.
Taken from [18]; the paper provides no independent validation that high IH corresponds to true rarity rather than label noise or outliers.
ad hoc to paper Oversampling instances with high prediction difficulty improves generalization performance on both rare and normal regions.
Core premise of InHaR-guided resampling; neither derived nor proven, only empirically tested.
domain assumption The regressor pool L used to compute hardness is representative enough that hardness values are stable and not simply in-sample overfitting.
Hardness is computed on the same model families later evaluated; no separate held-out estimation of hardness is described.
ad hoc to paper In the synthetic bimodal experiment, the 'rare regions' to be recovered are faithfully represented by high median IH per bin.
Paper labels top 30% IH bins as rare without an external ground-truth definition, making the validation circular (Section V-B).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Instance Hardness-Based Relevance for Imbalanced Regression." pith.science (2026). https://pith.science/paper/RJZLUE76

@misc{pith2026260720173,
  author       = {Pith},
  title        = {Pith review of: Instance Hardness-Based Relevance for Imbalanced Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RJZLUE76}},
  note         = {Machine review of arXiv:2607.20173}
}

read the original abstract

Imbalanced regression problems arise when the target variable has an asymmetric distribution, resulting in underrepresented value ranges in the dataset. Traditional approaches for identifying rare instances rely on a relevance function that assigns higher importance to specific regions of the target distribution. However, the effectiveness of imbalance-aware learning methods depends strongly on how relevance is defined. In more complex scenarios, such as bimodal distributions, traditional relevance functions struggle to capture rarity, as they assign fixed relevance values based solely on target values, thereby compromising the distinction between truly rare and normal instances. To address these limitations, this study proposes an Instance Hardness-based relevance function (InHaR) for identifying rare instances in regression problems. Unlike traditional relevance functions, the proposed approach incorporates learning difficulty, allowing rarity to be inferred not only from the target distribution but also from the difficulty of instances for the learning algorithm. This property is particularly important in bimodal scenarios, where rarity cannot be accurately inferred from target values alone. Experimental results demonstrate that the InHaR correctly identifies rare regions under bimodal distributions and, when used to guide resampling strategies such as Random Oversampling (RO) and Gaussian Noise (GN), leads to significant improvements in predictive performance compared to traditional relevance-based approaches. The code, dataset, and further details about the proposed method are publicly available at https://github.com/VitorLeitao/instance-hardness-Imbalanced-regression.

Figures

Figures reproduced from arXiv: 2607.20173 by the authors.

**Figure 3.** Identification of rare regions in a bimodal target distribution using [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 2.** Distribution of the correlation values between IH and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Behavior of the InHaR under increasing noise levels in a bimodal [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith

[1]

Learning from class-imbalanced data: Review of methods and applications,

H. Guo, Y . Li, J. Shang, M. Gu, Y . Huang, and B. Gong, “Learning from class-imbalanced data: Review of methods and applications,”Expert Systems with Applications, vol. 73, pp. 220–239, 2017

2017
[2]

Learning from imbalanced data: open challenges and future directions,

B. Krawczyk, “Learning from imbalanced data: open challenges and future directions,”Progress in Artificial Intelligence, vol. 5, no. 4, pp. 221–232, 2016

2016
[3]

Survey on deep learning with class imbalance,

J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,”Journal of Big Data, vol. 6, no. 1, pp. 1–54, 2019

2019
[4]

Ubl: an r package for utility- based learning,

P. Branco, R. P. Ribeiro, and L. Torgo, “Ubl: an r package for utility- based learning,”arXiv preprint arXiv:1604.08079, 2016

arXiv 2016
[5]

Imbalance in regression datasets,

D. Kowatsch, N. M. M ¨uller, K. Tscharke, P. Sperl, and K. B ¨otinger, “Imbalance in regression datasets,”arXiv preprint arXiv:2402.11963, 2024

arXiv 2024
[6]

Resampling strategies for imbalanced regression: a survey and empirical analysis,

J. G. Avelino, G. D. C. Cavalcanti, and R. M. O. Cruz, “Resampling strategies for imbalanced regression: a survey and empirical analysis,” Artificial Intelligence Review, vol. 57, no. 82, 2024

2024
[7]

Utility-based regression,

R. Ribeiro, “Utility-based regression,”Ph. D. dissertation, 2011

2011
[8]

Pre-processing approaches for imbalanced distributions in regression,

P. Branco, L. Torgo, and R. P. Ribeiro, “Pre-processing approaches for imbalanced distributions in regression,”Neurocomputing, vol. 343, pp. 76–99, 2019

2019

Show all 20 references

[9]

A comprehensive survey on imbal- anced regression: Definitions, solutions, and future directions,

S. Stocksieker and D. Pommeret, “A comprehensive survey on imbal- anced regression: Definitions, solutions, and future directions,” 2025, hAL open archive, hal-05213741

2025
[10]

Distance-based relevance function for imbalanced regression,

D. D. In and H. Kim, “Distance-based relevance function for imbalanced regression,”Stats, vol. 8, no. 3, p. 53, 2025

2025
[11]

Relating instance hardness to classification performance in a dataset: a visual approach,

P. Y . A. Paiva, C. C. Moreno, K. Smith-Miles, M. G. Valeriano, and A. C. Lorena, “Relating instance hardness to classification performance in a dataset: a visual approach,”Machine Learning, vol. 111, no. 8, pp. 3085–3123, 2022

2022
[12]

Smote for re- gression,

L. Torgo, R. P. Ribeiro, B. Pfahringer, and P. Branco, “Smote for re- gression,” inPortuguese conference on artificial intelligence. Springer, 2013, pp. 378–389

2013
[13]

Smogn: a pre-processing approach for imbalanced regression,

P. O. Branco, L. Torgo, and R. P. Ribeiro, “Smogn: a pre-processing approach for imbalanced regression,” inFirst International Workshop on Learning with Imbalanced Domains: Theory and Applications, vol. 74, 2017, pp. 36–50

2017
[14]

Geometric smote for regres- sion,

L. Camacho, G. Douzas, and F. Bacao, “Geometric smote for regres- sion,”Expert Systems with Applications, p. 116387, 2022

2022
[15]

Imbalanced regression and extreme value prediction,

R. P. Ribeiro and N. Moniz, “Imbalanced regression and extreme value prediction,”Machine Learning, vol. 109, pp. 1803–1835, 2020

2020
[16]

Nonnegativity- , monotonicity-, or convexity-preserving cubic and quintic hermite interpolation,

R. L. Dougherty, A. S. Edelman, and J. M. Hyman, “Nonnegativity- , monotonicity-, or convexity-preserving cubic and quintic hermite interpolation,”Mathematics of Computation, vol. 52, no. 186, pp. 471– 494, 1989

1989
[17]

Exploratory data analysis, limited prelim. ed,

J. Tukey, “Exploratory data analysis, limited prelim. ed,” 1970

1970
[18]

Instance hardness measures for classification and regression problems,

G. P. Torquette, V . S. Nunes, P. Y . A. Paiva, and A. C. Lorena, “Instance hardness measures for classification and regression problems,” Instituto Tecnol´ogico de Aeron ´autica (ITA), S ˜ao Jos ´e dos Campos, SP, Brazil, Tech. Rep., 2024, published: 27 February 2024

2024
[19]

Utility-based regression,

L. Torgo, R. P. Ribeiro, and B. Pfahringer, “Utility-based regression,” inLecture Notes in Computer Science. Springer, 2009, vol. 5812

2009
[20]

Precision and recall for regression,

L. Torgo and R. Ribeiro, “Precision and recall for regression,” in International Conference on Discovery Science. Springer, 2009, pp. 332–346

2009

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Learning from class-imbalanced data: Review of methods and applications,

H. Guo, Y . Li, J. Shang, M. Gu, Y . Huang, and B. Gong, “Learning from class-imbalanced data: Review of methods and applications,”Expert Systems with Applications, vol. 73, pp. 220–239, 2017

2017

[2] [2]

Learning from imbalanced data: open challenges and future directions,

B. Krawczyk, “Learning from imbalanced data: open challenges and future directions,”Progress in Artificial Intelligence, vol. 5, no. 4, pp. 221–232, 2016

2016

[3] [3]

Survey on deep learning with class imbalance,

J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,”Journal of Big Data, vol. 6, no. 1, pp. 1–54, 2019

2019

[4] [4]

Ubl: an r package for utility- based learning,

P. Branco, R. P. Ribeiro, and L. Torgo, “Ubl: an r package for utility- based learning,”arXiv preprint arXiv:1604.08079, 2016

arXiv 2016

[5] [5]

Imbalance in regression datasets,

D. Kowatsch, N. M. M ¨uller, K. Tscharke, P. Sperl, and K. B ¨otinger, “Imbalance in regression datasets,”arXiv preprint arXiv:2402.11963, 2024

arXiv 2024

[6] [6]

Resampling strategies for imbalanced regression: a survey and empirical analysis,

J. G. Avelino, G. D. C. Cavalcanti, and R. M. O. Cruz, “Resampling strategies for imbalanced regression: a survey and empirical analysis,” Artificial Intelligence Review, vol. 57, no. 82, 2024

2024

[7] [7]

Utility-based regression,

R. Ribeiro, “Utility-based regression,”Ph. D. dissertation, 2011

2011

[8] [8]

Pre-processing approaches for imbalanced distributions in regression,

P. Branco, L. Torgo, and R. P. Ribeiro, “Pre-processing approaches for imbalanced distributions in regression,”Neurocomputing, vol. 343, pp. 76–99, 2019

2019