Machine Learning for Exoplanet Detection: A Comparative Analysis Using Kepler Data

Fatemeh S. Tabatabaei; Mahdiyar Mousavi-Sadr; Mohammad H. Zhoolideh Haghighi; Reihaneh Karimi

Machine Learning for Exoplanet Detection: A Comparative Analysis Using Kepler Data

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a Random Forest classifier trained on Kepler flux time series can identify exoplanet candidates with 99.8% accuracy, outperforming KNN, Decision Tree, and Logistic Regression.

desk verdict A routine classifier benchmark on Kepler data whose headline 99.8% accuracy is plausible but unverifiable from the abstract alone. read the letter →

arxiv 2508.09689 v1 pith:DBMW5QJC submitted 2025-08-13 astro-ph.EP astro-ph.IMphysics.comp-phphysics.data-an

Reihaneh Karimi , Mahdiyar Mousavi-Sadr , Mohammad H. Zhoolideh Haghighi , Fatemeh S. Tabatabaei This is my paper · ORCID

classification astro-ph.EPastro-ph.IMphysics.comp-phphysics.data-an

keywords exoplanetdetectionmachinelearningRandomForestKeplerphotometrySMOTEclassificationtransitsignalsimbalanceddata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a supervised machine-learning pipeline using Kepler photometric time series can automatically flag exoplanet candidates, and that Random Forest is the most accurate and robust of four standard classifiers. On 5,074 stars with 3,198 flux measurements each, the authors report Random Forest accuracy of 99.8%, KNN at 99.3%, Decision Tree at 97.1%, and Logistic Regression at 95.8%. They also report that SMOTE balancing improves every model, and they argue the resulting pipeline is suitable for automated screening of large photometric datasets at ground-based observatories. A reader should care because accurate automated triage could shorten the path from light curves to exoplanet candidates.

What carries the argument

The carrying mechanism is the Random Forest ensemble, a collection of decision trees whose averaged votes reduce individual-tree overfitting, applied to per-star flux measurements and combined with the Synthetic Minority Over-sampling Technique (SMOTE), which creates synthetic samples of the rare planet class to rebalance training data. The Random Forest does the classification; SMOTE does the imbalance correction; together they produce the reported 99.8% accuracy.

What would settle it

Reproduce the pipeline on the same 5,074 stars with a star-level split in which SMOTE is applied after splitting and no star's flux measurements appear in both training and test. If a majority-class baseline that always predicts 'non-planet' matches the reported accuracies, or if Random Forest's accuracy drops materially below 99.8%, the central claim collapses.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that an ensemble tree classifier is the strongest of four standard algorithms for classifying Kepler photometric time series as exoplanet candidates versus non-candidates. The authors attribute Random Forest's lead, reported as 99.8% accuracy with near-perfect F1-scores, to superior generalization and robustness. KNN is close, Decision Tree is moderate, and Logistic Regression trails, which the authors read as evidence that flexible nonlinear methods match the structure of this data better than linear separators. They further claim that SMOTE's consistent accuracy gains show class imbalance was the main bottleneck.

Load-bearing premise

The reported numbers stand or fall on a leakage-free evaluation: the train/test split must separate stars, SMOTE must be applied only to the training partition, and the planet/non-planet labels must be correct; otherwise 99.8% can reflect class imbalance or memorization rather than predictive skill.

Editorial extensions

If this is right

If the 99.8% accuracy holds, Random Forest can serve as a reliable first-pass screening tool for exoplanet candidates in Kepler-sized photometric surveys.
SMOTE's accuracy gains across all four classifiers indicate that imbalanced labels were suppressing performance and that oversampling is a workable fix for this type of data.
The reported ordering Random Forest > KNN > Decision Tree > Logistic Regression implies nonlinear ensemble methods are better suited than linear models to flux time series classification.
The pipeline's dependence on only flux measurements keeps it lightweight enough for automated processing of large catalogs at ground-based facilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A fair external test would withhold entire observing quarters or campaigns rather than random stars, because flux time series from the same instrument settings share systematic noise that random splits can leak across.
Re-running the same four classifiers on TESS light curves or on synthetic transits injected into quiet stars would show whether 99.8% transfers to other surveys or is Kepler-specific.
Because KNN is only a small margin behind Random Forest, the reported lead could depend on the particular SMOTE-generated training set; repeated SMOTE with different random seeds would reveal whether the Random Forest advantage is stable.
A majority-class baseline should be reported alongside the four classifiers: if a constant 'non-planet' predictor already exceeds 95% accuracy on the same test split, the absolute accuracies would become less informative.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a straightforward comparative benchmark: four off-the-shelf classifiers, one public Kepler dataset, standard metrics, and a clear deployment target (INO). There is no new method, no new data, and no claim that changes the physics. The likely result, that random forest beats logistic regression on separable photometric features, is well established in the prior ML-exoplanet literature. So the novelty is modest at best.

What the paper does well, from the abstract, is use a public dataset, report multiple standard metrics, and explicitly frame the work as a practical screening pipeline. That framing makes the comparison potentially useful as an engineering benchmark, not as a scientific contribution.

The soft spots are the usual ones for this genre, but they matter. The abstract gives no train/test split, no cross-validation scheme, no class counts, no error bars, and no description of whether SMOTE was applied before splitting. The stress-test concern is on target: if SMOTE touched the full dataset before the split, synthetic minority instances can leak into the test set and inflate the numbers. Even without that, a heavily imbalanced sample can make accuracy an uninformative metric, and the abstract shows no constant baseline or confusion-matrix numbers. The "superior generalization" claim is simply unsupported at this level of description. None of this makes the work fraudulent; it just means the central number cannot be assessed from what is currently visible.

The full text may well contain the missing evaluation details, and if it does, the paper could serve as a useable reference for teams deploying automated transit screening at smaller observatories. But as it stands, the contribution is a modest benchmark that needs referees to audit the evaluation protocol before anyone should trust the 99.8% figure.

My take: this deserves a serious peer review, not a desk reject, because the question of whether the evaluation is leakage-free is exactly what referees should check. But I would not cite it in my own work unless the full text shows rigorous splitting and error reporting. It is not a reading group pick for us; it would only be useful to someone specifically building a transit-screening pipeline from scratch.

Referee Report

4 major / 3 minor

Summary. The manuscript describes a supervised machine-learning benchmark for exoplanet detection using Kepler photometry, comparing Random Forest, KNN, Decision Tree, and Logistic Regression, and reporting accuracies of 99.8%, 99.3%, 97.1%, and 95.8%, respectively, with SMOTE improving performance. The authors claim Random Forest shows superior generalization and propose deployment at INO. However, the abstract provides no details of the evaluation protocol, data splitting, or class balance, making the advertised accuracies unverifiable at this level.

Significance. If the reported accuracies reflect a leakage-free, out-of-sample evaluation, the paper would provide a useful benchmark for automatic exoplanet screening and a credible demonstration that ensemble methods outperform simple linear classifiers on this feature set. The study is potentially relevant to ground-based facilities like INO. The abstract is explicit about dataset size and algorithms, and the ranking is plausible. However, the missing methodological detail prevents assessment of the central claim; the paper's contribution is currently only a collection of performance numbers without evidence of generalization. No code, data, or machine-checked proofs are mentioned.

major comments (4)

[Abstract, first paragraph] The 99.8% accuracy for Random Forest is the central result, but the abstract does not specify any train/test split, cross-validation procedure, or holdout set. Without this, the number could arise from leakage, memorization, or evaluation on the training set. Please report the exact split ratio, number of folds, and whether the reported metric is the mean over folds.
[Abstract, second paragraph] The statement that SMOTE 'significantly improves performance' is ambiguous regarding where SMOTE is applied. If synthetic samples are generated before splitting, the test set contains synthetic points or near-duplicates of training data, inflating accuracy and F1. State explicitly that SMOTE was applied only to the training partition (or within each cross-validation fold).
[Abstract, second paragraph] The abstract mentions class imbalance but does not report the class distribution. In a sample of 5,074 stars, if planet candidates are a small minority, a trivial majority-class classifier can exceed 90% accuracy. Provide the planet/non-planet ratio, the confusion matrix (which is listed among metrics but no values are given), and the accuracy of a constant baseline.
[Abstract, first paragraph] The unit of analysis is unclear: '3,198 flux measurements across 5,074 stars' could mean 5,074 rows each with 3,198 features, or a total of 3,198 measurements over all stars. If rows are individual measurements, random splitting without grouping by star can place temporally correlated observations from the same star into both train and test, making the model memorize star-specific noise. Clarify the feature matrix dimensions and the split's grouping level.

minor comments (3)

[Abstract, second paragraph] 'near-perfect F1-scores' should be quantified; report per-class precision/recall for the minority class.
[Abstract, first paragraph] 'comprehensive set of evaluation metrics' includes learning curves, but the abstract does not state what the learning curves show; a one-sentence summary would help.
[Abstract, second paragraph] The claim 'superior generalization' is not supported by a statistical comparison (e.g., confidence intervals or significance tests across folds); add such evidence or temper the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; abstract reports a standard ML benchmark with no derivation chain or self-citation.

full rationale

The abstract describes a supervised machine learning comparison on Kepler photometry. There is no formal derivation, no fitted parameter being relabeled as a prediction, and no self-citation invoked to justify the central claim. The reported accuracies are properties of models trained and tested on the dataset; this is inherent to any empirical ML benchmark and does not constitute circularity. The concern about potential data leakage or improper SMOTE placement is a question of experimental validity, not logical circularity. Without access to the full methods, we cannot identify any equation or construction that reduces to its own inputs. Therefore, the paper is not circular at the level of the abstract.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central claims rest on the evaluation protocol, which the abstract does not describe. Six categories of hand-set or tuned settings, the classifier hyperparameters, SMOTE configuration, and split choice, determine the headline accuracies. Four domain assumptions must hold for the numbers to mean what the abstract says: correct labels, a leakage-free split, sufficient features, and faithful SMOTE synthesis.

free parameters (6)

Random Forest hyperparameters (n_estimators, max_depth, min_samples_split)
Set by hand or tuned; directly determine the headline 99.8% accuracy.
KNN k and distance metric
Determine KNN's 99.3% accuracy.
Decision Tree split criterion and pruning
Determine DT's 97.1% accuracy.
Logistic Regression regularization strength
Determines LR's 95.8% accuracy.
SMOTE sampling strategy and k_neighbors
Abstract credits SMOTE for improving all models; its settings affect every reported score.
Train/test split ratio and random seed
Reported generalization scores depend entirely on this choice; abstract does not state it.

assumptions (4)

domain assumption Class labels for the 5,074 stars are correct (confirmed Kepler exoplanet candidates).
Accuracy claims are only meaningful if the training labels are true; abstract does not describe label provenance.
domain assumption Train/test split is leakage-free and SMOTE is applied only to the training partition.
99.8% accuracy on imbalanced data is only interpretable as generalization if no test information reaches training; abstract describes no split.
domain assumption The 3,198 flux measurements per star, after preprocessing, define a learnable separation between planets and non-planets.
No feature engineering is described; the classifiers' success depends entirely on these features.
domain assumption SMOTE-synthesized minority samples are representative of the real planet-star distribution.
If synthetic points distort the decision boundary, reported accuracies are optimistic.

invented entities (1)

none
purpose: No new entities postulated
The paper works entirely within standard supervised classification and introduces no new physics beyond the Kepler data itself.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Machine Learning for Exoplanet Detection: A Comparative Analysis Using Kepler Data." pith.science (2026). https://pith.science/paper/DBMW5QJC

@misc{pith2026250809689,
  author       = {Pith},
  title        = {Pith review of: Machine Learning for Exoplanet Detection: A Comparative Analysis Using Kepler Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBMW5QJC}},
  note         = {Machine review of arXiv:2508.09689}
}

read the original abstract

The discovery of exoplanets has expanded our understanding of planetary systems and opened new avenues for astronomical research. In this study, we present a machine learning (ML) framework for exoplanet identification using a time-series photometric dataset from the Kepler Space Telescope, comprising 3,198 flux measurements across 5,074 stars. We investigate the performance of four supervised classification algorithms, namely Random Forest, k-Nearest Neighbors (KNN), Decision Tree, and Logistic Regression, using a comprehensive set of evaluation metrics such as accuracy, precision, recall, F1-score, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), confusion matrices, and learning curves. Among the models, Random Forest achieves the highest accuracy (99.8\%) and near-perfect F1-scores, demonstrating superior generalization and robustness. KNN also performs strongly, achieving 99.3\% accuracy, while Decision Tree demonstrates moderate performance with 97.1\% accuracy, and Logistic Regression trails behind with the lowest accuracy and generalization at 95.8\%. Notably, the application of the Synthetic Minority Over-sampling Technique (SMOTE) significantly improves performance across all models by addressing class imbalance. These findings underscore the effectiveness of ensemble-based machine learning techniques, particularly Random Forest, in handling large volumes of photometric data for automated exoplanet detection. This approach holds significant potential for implementation at ground-based facilities, such as the Iranian National Observatory (INO), where such extensive and precise datasets can further advance exoplanet discovery and characterization efforts.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.