Improving Random Forests by Smoothing

Daniel F. Schmidt; Mario Boley; Phuc Luong; Ziyi Liu

arxiv: 2505.06852 · v2 · pith:MXSIITGNnew · submitted 2025-05-11 · 💻 cs.LG · stat.ML

Improving Random Forests by Smoothing

Ziyi Liu , Phuc Luong , Mario Boley , Daniel F. Schmidt This is my paper

Pith reviewed 2026-05-22 15:38 UTC · model grok-4.3

classification 💻 cs.LG stat.ML

keywords random forestkernel smoothingregressionsmall sampleensemble methodsnonparametric regressionpredictive performance

0 comments

The pith

Kernel smoothing of random forest outputs captures split-point variability and improves accuracy in small-data regimes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Random forest regression builds piecewise constant predictions by recursively partitioning the input space, with each region receiving an independent average. This paper adds a kernel smoothing step after the forest is built, so that each prediction becomes a locally weighted average of nearby forest outputs. The procedure is presented as a way to reflect the fact that different resamples of the training data would have produced slightly different cut points. Readers would care because the added local regularity lets information flow across adjacent partitions, which reduces error when the training set is too small for each partition to be estimated reliably on its own.

Core claim

The central claim is that post-processing the piecewise constant outputs of a random forest with kernel smoothing captures the variability in tree cut points that arises under resampling of the training inputs and thereby reduces mean squared error, with the improvement being largest when the number of training observations is limited.

What carries the argument

Kernel smoothing operator applied directly to the random forest's piecewise constant function values, which averages predictions over local neighborhoods in the input space.

If this is right

Predictive performance improves on a range of regression tasks relative to standard random forests.
The largest gains occur in the small-sample setting where each partition would otherwise be estimated from very few points.
The adaptive partitioning property of random forests is retained while local regularity is added.
The smoothing step admits an interpretation as accounting for uncertainty in the locations of the tree splits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same post-processing idea might be tried on other recursive partitioning methods such as single decision trees or gradient-boosted trees.
In very high-dimensional inputs the kernel would need to be chosen or scaled carefully to keep the local averaging meaningful.
An experiment that varies the degree of smoothness of the target function would map out the regime where the method helps versus where it hurts.

Load-bearing premise

The underlying regression function is locally regular enough that averaging nearby forest outputs reduces variance more than any bias introduced by crossing true discontinuities.

What would settle it

On synthetic data drawn from a step function with many sharp jumps, the smoothed forest would have to show higher error than the unsmoothed forest for the performance claim to be refuted.

read the original abstract

Random forest regression is a powerful non-parametric method that adapts to local data characteristics through data-driven partitioning, making it effective across diverse application domains. However, the piecewise constant nature of random forest predictions means each partition is predicted independently, ignoring potential smoothness in the underlying function. Particularly in the small data regime, this lack of information sharing across the input space can lead to suboptimal performance. In this work, we propose a kernel-based smoothing mechanism that enhances random forests by introducing local regularity to their predictions while preserving their adaptive partitioning capabilities. Our approach applies kernel smoothing to the piecewise constant outputs of random forests, effectively combining the adaptability of tree-based methods with the smoothness assumptions of kernel methods. We show that this smoothing procedure can be interpreted as capturing the variability/uncertainty in the tree cut points under resampling of the training inputs. Empirical results demonstrate that the proposed smoothed random forest model consistently improves predictive performance across diverse test cases, particularly in data-scarce settings. Code, datasets, and experiment results are publicly available at https://github.com/Neal-Liu-Ziyi/SmoothedRandomForest.git.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Kernel smoothing on random forest outputs shows small-sample gains but the cut-point variability interpretation does not follow from the described operations.

read the letter

This paper's key idea is to smooth the piecewise constant predictions from a random forest using a kernel, which helps in low-data situations by sharing information across nearby regions. The work does well by making the code and data public, allowing easy checks on the reported gains. The method is lightweight and doesn't require changing the forest training itself. On the positive side, the abstract indicates consistent improvements across diverse tests, particularly when samples are scarce. This could appeal to applied users who rely on random forests but notice they overfit or vary too much with limited data. The soft spots center on the proposed mechanism. The claim that smoothing captures uncertainty in cut points from resampling doesn't follow directly from the operations described. Smoothing a single forest's output averages response values on a fixed partition, whereas the ensemble already averages across varied partitions. These are different unless the partitions are similar or the function is smooth locally, and the paper doesn't show this equivalence holds in general. The gains might therefore come from ordinary smoothing rather than the specific interpretation given. Experiments would benefit from more transparency on bandwidth selection, baseline comparisons, and significance of the improvements. This is useful for practitioners in fields with small datasets who want to enhance their random forest models without much extra effort. Readers working on tree ensembles or kernel methods might find the practical angle worth considering. I would recommend putting it through peer review. The reproducibility via the GitHub repo makes it straightforward for referees to evaluate the claims and suggest improvements to the justification.

Referee Report

1 major / 2 minor

Summary. The manuscript proposes applying post-hoc kernel smoothing to the piecewise-constant regression outputs of a standard random forest. The central claim is that this smoothing can be interpreted as capturing the variability in tree cut-points induced by resampling the training inputs, thereby introducing local regularity while retaining the forest's adaptive partitioning. Empirical results are reported to show consistent predictive gains over unsmoothed random forests, especially in small-sample regimes.

Significance. If the interpretation and performance claims hold, the work supplies a lightweight, interpretable post-processing step that blends the strengths of tree-based partitioning with kernel smoothness. Public release of code, datasets, and results supports reproducibility and allows direct verification of the reported gains.

major comments (1)

Abstract and the section presenting the resampling interpretation: the claim that kernel smoothing of the fixed forest output mathematically captures (or approximates) the variability of cut-points under resampling of the training inputs is load-bearing for the bias-variance justification. The random forest already averages step functions whose discontinuity locations vary across bootstrap samples; smoothing the single ensemble output instead averages response values on a fixed partition. No derivation or set of assumptions (e.g., small cut-point jitter relative to bandwidth, local Lipschitz continuity) is supplied showing when or why these two operations coincide, leaving open the possibility that observed gains arise from generic smoothing rather than the stated mechanism.

minor comments (2)

Experimental details: the manuscript should specify the exact procedure for selecting the kernel bandwidth (cross-validation, rule-of-thumb, or fixed), the number of repetitions for statistical testing, and whether the same hyper-parameters were used for the baseline random forest and the smoothed variant.
Notation and presentation: ensure the kernel function and bandwidth parameter are introduced with consistent symbols and that any figures comparing smoothed vs. unsmoothed predictions clearly label the bandwidth value used.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thoughtful and detailed review. The feedback highlights an important point about the strength of the resampling interpretation, and we address it directly below.

read point-by-point responses

Referee: Abstract and the section presenting the resampling interpretation: the claim that kernel smoothing of the fixed forest output mathematically captures (or approximates) the variability of cut-points under resampling of the training inputs is load-bearing for the bias-variance justification. The random forest already averages step functions whose discontinuity locations vary across bootstrap samples; smoothing the single ensemble output instead averages response values on a fixed partition. No derivation or set of assumptions (e.g., small cut-point jitter relative to bandwidth, local Lipschitz continuity) is supplied showing when or why these two operations coincide, leaving open the possibility that observed gains arise from generic smoothing rather than the stated mechanism.

Authors: We agree that the current presentation of the resampling interpretation would be strengthened by greater precision and supporting discussion. The manuscript frames the kernel smoothing as an interpretive device that can be viewed as capturing variability in cut-points induced by bootstrap resampling of the training data, motivated by the fact that individual trees are trained on different samples and thus exhibit different partitions. However, we acknowledge that no formal derivation is provided showing when or under what conditions smoothing the fixed ensemble output approximates averaging over an ensemble of resampled partitions. In the revised version we will (i) qualify the language in the abstract and the relevant section to describe the connection as an intuitive interpretation rather than a strict mathematical equivalence, and (ii) add a short discussion of plausible assumptions (small cut-point jitter relative to bandwidth, local regularity of the target function) under which the two procedures can be expected to produce similar effects. We will also note that the empirical gains are consistent with this view but do not by themselves prove the mechanism. These changes will make the justification more transparent and help distinguish the proposed approach from generic post-hoc smoothing. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical gains rest on independent smoothing application, not self-referential derivation

full rationale

The paper introduces kernel smoothing applied post-hoc to the piecewise-constant outputs of a trained random forest and offers an interpretive claim that this procedure captures cut-point variability under resampling. This interpretation is not derived via equations that reduce the smoothed predictor to an average over resampled trees by construction, nor is it justified by any self-citation to a uniqueness theorem or prior ansatz from the same authors. Performance improvements are reported through direct empirical evaluation on diverse datasets rather than through any fitted parameter that is then relabeled as a prediction. The central mechanism therefore remains an independent modeling choice whose validity is tested externally, satisfying the criteria for a self-contained derivation chain.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The method rests on the domain assumption that the target function is locally regular enough for kernel smoothing to be beneficial, plus the modeling choice that the kernel bandwidth can be chosen without introducing new free parameters that dominate the result. No new physical entities are postulated.

free parameters (1)

kernel bandwidth
The smoothing kernel requires a bandwidth parameter whose value is not derived from first principles and must be selected or cross-validated on data.

axioms (1)

domain assumption The regression function possesses sufficient local regularity for kernel smoothing to reduce mean squared error.
Invoked when the authors claim that smoothing captures variability in tree cut points and improves performance in small-data regimes.

pith-pipeline@v0.9.0 · 5720 in / 1312 out tokens · 29577 ms · 2026-05-22T15:38:08.343041+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose to transform this to a smooth (differentiable) function by smoothing f itself; ... ˆ˜y(x0 | X, y, λ) = β1 ˆy(x0 | X, y, λ) + β0 (eq. 1); ... P(z ∈ Di | x0, λ) (eq. 2)
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Theorem 1 ... n(ˆb − b) → w L ... Laplace distribution ... mimic the effect that resampling ... has on the estimated breakpoints

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.