{"id":"ae0f0b7f-9bc7-4e8a-9686-028a139089ea","arxiv_id":"2505.15600","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A variational quantum classifier reaches 0.95 accuracy on pulsar candidate classification in its best configuration, but the evaluation is not statistically rigorous and has no classical comparison.","lead":"Scientists applied a quantum machine learning tool, the Variational Quantum Classifier, to the HTRU-2 pulsar candidate dataset and report top accuracies of 0.95 with certain circuit choices. A smart generalist should read it to see how quantum classifiers are being benchmarked on real astronomy data, but the study lacks error bars, classical baselines, and a rigorous holdout procedure.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table VII's headline metrics are internally inconsistent: at HTRU-2's stated 9.8% pulsar rate, accuracy 0.950 with precision 0.375 and recall 0.750 is arithmetically impossible, so the central 0.950 claim lacks a coherent evaluation setup.","rationale":"I read the paper's central claim as the statement that a Qiskit VQC configuration reaches 0.950 accuracy on HTRU-2 pulsar classification, with best MCC 0.670. The reader's rejection is based on test-set selection and possible feature-selection leakage. Those concerns are valid, but I found a more basic and more decisive problem: the aggregate metrics in Table VII are internally inconsistent with the dataset's stated class imbalance. This can be checked by simple algebra from the definitions in Appendix A, before any question of data snooping arises. For the headline row, the reported precision and recall force the false-positive and false-negative rates to make accuracy at most about 0.853 if the test set inherits the HTRU-2 prevalence. Similar arithmetic fails for most high-recall, low-precision rows. The only ways to resolve the contradiction are an undisclosed rebalanced test set, different test sets for different rows, or numerical errors. Under any of these outcomes, the 0.950 figure is not a trustworthy generalization estimate on the HTRU-2 distribution. I therefore keep the reader's REJECT verdict, although my load-bearing concern is not the one the reader identified as weakest. The proposed rerun with a full confusion matrix would settle the issue definitively.","tokens_in":10637,"tokens_out":7162,"duration_ms":63114,"concrete_test":"Rerun the exact Table VII row 1 configuration (FS1, 3 features, ZZ feature map, EfficientSU2 ansatz, circular entanglement, 180 training samples) using the same train/test split code, and output the full confusion matrix plus the test-set class distribution. If the test set is the remaining HTRU-2 data (p about 0.098), compute Acc = 1 - p[R(1-P)/P + (1-R)]; a value near 0.950 would require p about 0.033 and would reveal undisclosed filtering or rebalancing. Also check whether the other Table VII rows were evaluated on different test subsets; the raw predictions will settle which row, if any, reflects the actual experimental setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim relies on Table VII. Its first row reports accuracy 0.950, precision 0.375, recall 0.750 for a 3-feature FS1 model trained on 180 samples. Section II A states that only 9.8% of the dataset are pulsars, and the paper does not state that the test set was rebalanced. Let p be the test-set pulsar fraction, R recall, and P precision. From TP = pRN, FP = TP(1-P)/P, and FN = p(1-R)N, the implied accuracy is Acc = 1 - p[R(1-P)/P + (1-R)]. With p = 0.098, R = 0.75, P = 0.375, this gives Acc = 0.853, not 0.950. The contradiction repeats across the table: row 2 (Acc 0.945, P 0.476, R 1.000) implies Acc 0.892, and row 3 (Acc 0.940, P 0.429, R 1.000) implies Acc 0.870. These entries cannot all come from one test set with the stated class imbalance unless the test set was filtered, rebalanced, or metrics were computed on different subsets without disclosure. Because the conclusion's 'accuracies as high as 0.950' is taken directly from this table, the central claim is unsupported by the reported numbers as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript applies Qiskit's Variational Quantum Classifier (VQC) to the HTRU-2 pulsar candidate dataset. It normalizes data with MinMaxScaler, compares two feature-selection orderings (FS1 via SelectKBest, FS2 via correlation with the class), varies the number of features/qubits between 2 and 8, training sample sizes of 180 and 300, feature maps (ZZ and Pauli), ansatze (Real Amplitudes and EfficientSU2), and entanglement patterns. Performance is reported via accuracy, precision, recall, F1, and MCC. The central conclusion is that the VQC is easy to use and performs well, with accuracies as high as 0.950 and a best MCC of 0.670.","tokens_in":10932,"tokens_out":6068,"duration_ms":54367,"significance":"The empirical question addressed is of some interest for QML applications in astrophysics, and the manuscript is honest about the class imbalance by reporting MCC alongside accuracy. Its strength is a fairly systematic sweep of circuit components and feature-selection choices for a specific Qiskit implementation. However, the contribution is incremental relative to existing QML pulsar classification studies, and the central empirical claim is currently unsupported: the headline numbers in Table VII are internally inconsistent, the model selection procedure uses test performance, no classical baseline is given, and no code, seeds, or repeated runs are provided. If the experiments were rerun with proper validation and baselines, the paper could be a useful application note, but as it stands the reported 'good performance' is not a reliable measurement.","major_comments":[{"comment":"The headline accuracy entries are arithmetically inconsistent with the precision, recall, and the dataset's stated 9.8% pulsar rate. For a test set with pulsar fraction p, the metrics imply Acc = 1 - p[R(1-P)/P + (1-R)]. With p=0.098, row 1 (Acc 0.950, P 0.375, R 0.750) gives 0.853; row 2 (Acc 0.945, P 0.476, R 1.000) gives 0.892; and row 3 (Acc 0.940, P 0.429, R 1.000) gives 0.870. The paper does not report any rebalancing or filtering of the test set, and Section II A describes the data as having 9.8% pulsars. Either the accuracy or the other metrics are not from the same evaluation, and the Conclusion's 'accuracies as high as 0.950' is therefore unsupported by the reported numbers.","section":"Section IV, Table VII"},{"comment":"The reported values are selected maxima over a large grid of configurations, with no validation set and no repeated runs. The text states that some of the 'best-performing (BP) models, based on accuracy' were selected and that Table VII entries whose counterpart 'was not high enough' were 'filtered out'. Selecting on the test set makes the headline accuracy an estimate of the search procedure, not of generalization. The authors should report held-out or nested-validation performance with means and standard deviations over repeated runs, or the claim of 'good performance' cannot be evaluated.","section":"Section IV, Figs. 7-10 and Table VII"},{"comment":"It is not stated whether the FS1 and FS2 feature-selection procedures were applied before or after the train/test split. If the feature ranking was computed on the full dataset, class labels of the test set influenced the preprocessing, and all downstream numbers are optimistically biased. This must be clarified and, if leakage exists, the experiments must be rerun with feature selection nested inside the training folds.","section":"Section II C"},{"comment":"No classical baseline is reported for the same data and preprocessing, and no comparison is made to the HTRU-2 results in the cited literature or to the QML pulsar studies cited as Refs. [29,30]. Without a baseline, 'good performance' is not a supported claim: on a dataset with 9.8% positives, a degenerate classifier can already reach roughly 0.902 accuracy, so accuracy alone is insufficient evidence. A comparison against standard classical classifiers (e.g., logistic regression, random forest, SVM) on the same train/test protocol is needed.","section":"Section IV and Conclusion"}],"minor_comments":[{"comment":"'HRTU-2' should be 'HTRU-2'.","section":"Section II A"},{"comment":"'representend' should be 'represented', and 'est performing' should be 'best performing'.","section":"Fig. 8 and Fig. 10 captions"},{"comment":"BP-A and BP-B list identical configurations (ZZ feature map, Real Amplitudes, linear entanglement); if this is not a typo, the labels cannot distinguish the two curves in Fig. 7.","section":"Table III"},{"comment":"No Qiskit version, random seeds, or code availability are reported, which limits reproducibility of the single-run results.","section":"General"},{"comment":"The notation 'EfficientSU2' is written without a space in some rows; consistent naming would improve readability.","section":"Table VII"}],"recommendation":"reject","confidential_remarks":"For the editor: the manuscript has the structure of an application note, but the main empirical claim fails basic internal consistency checks. I would regard a resubmission as viable only if it includes a corrected evaluation protocol (nested feature selection, validation-based model selection, repeated runs with seeds, and a classical baseline) and resolves the Table VII arithmetic. The paper's current form is not suitable for publication in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the Qiskit VQC pulsar paper. Bottom line: it's a straightforward application of a standard QML algorithm to a public dataset, and it does that clearly enough, but the headline result doesn't survive contact with its own numbers.\n\nWhat's new: it's the first VQC run on HTRU-2, as far as the cited literature shows. The paper compares two feature-selection methods, different feature maps and ansatze, and reports accuracy, precision, recall, F1, and MCC. That's a legitimate extension of the prior QCNN/QSVM work, and the circuit descriptions are reasonably clear.\n\nWhat it does well: the writing is organized; the choice of MCC for an imbalanced dataset is sensible; the setup of the feature maps and ansatze is standard and reproducible from the text.\n\nThe soft spots are severe, not minor. First, the evaluation is not trustworthy. The paper selects best-performing models from a grid using test-set accuracy and then filters out configurations that don't make the cut for display. That's data snooping. There are no error bars or repeated seeds. No classical baseline is provided, so even the honest numbers don't tell us whether VQC adds anything.\n\nSecond, and more decisively, Table VII is internally inconsistent. With the stated 9.8% pulsar fraction in the test set, the first row's accuracy 0.950, precision 0.375, recall 0.750 is arithmetically impossible; the implied accuracy is about 0.853. The same contradiction repeats across the table. Unless the test set was rebalanced or the metrics were computed on different subsets, and the paper says nothing like that, the reported numbers cannot all be true. The conclusion's \"accuracies as high as 0.950\" comes straight from this table, so the central claim hangs on numbers that don't add up.\n\nCitation pattern: fine. The relevant QML pulsar papers are cited, and the HTRU-2 reference is there. Self-citation isn't an issue.\n\nWho is this for? Someone curious about whether VQC can touch this dataset might skim it. But as it stands, the paper doesn't support its own conclusion. A serious referee would need to send the authors back for a redesigned evaluation: proper nested model selection, multiple seeds, classical baselines, and a corrected table. That's a substantial rewrite.\n\nMy recommendation: desk reject. The internal inconsistency alone is disqualifying, and the contribution is a routine application with no demonstrated advantage. If the authors fix the evaluation, the paper might become a modest data point, but that's a different manuscript.","headline":"Routine QML application to HTRU-2 whose headline accuracy is arithmetically impossible given the stated class imbalance; the reported numbers don't add up.","tokens_in":11492,"tokens_out":2930,"would_cite":false,"duration_ms":23103,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Qiskit variational quantum classifier reaches 95 percent accuracy on pulsar candidates from the HTRU-2 survey.","keywords":["quantum machine learning","variational quantum classifier","pulsar classification","HTRU-2","feature selection","Qiskit","imbalanced classification"],"falsifier":"Rerun the three-feature VQC configuration with feature selection fitted inside each training fold of a cross-validation loop on the full HTRU-2 dataset, and compare the held-out accuracy with the reported 0.950. If the cross-validated accuracy falls materially below 0.950, the headline number came from leakage or from selecting the best of many configurations on the test set.","tokens_in":10424,"feed_emoji":"📡","tokens_out":9447,"duration_ms":74785,"temperature":0.7,"pith_summary":"Quantum machine learning is still young, and this paper asks whether a ready-made variational circuit can handle a real astronomy task: separating pulsars from non-pulsars in the HTRU-2 candidate dataset, where only 9.8 percent of examples are true pulsars. The paper reports that the Qiskit Variational Quantum Classifier reaches an accuracy of 0.950 with three features, using the ZZ feature map and the EfficientSU2 ansatz, with the best Matthews correlation coefficient reaching 0.670. The point is constructive: a small three-qubit hybrid circuit is easy to deploy and can perform well on an imbalanced astrophysical dataset, but the choice of feature selection, data encoding, ansatz, and entanglement pattern matters measurably. The conclusion also notes that using the same circuit as both feature map and ansatz yields high precision at the cost of much lower accuracy.","feed_headline":"Quantum classifier hits 95 percent on pulsar candidates","feed_subtitle":"A three-qubit Qiskit VQC reaches 0.950 accuracy on HTRU-2; best MCC is 0.670.","key_machinery":"The machinery is the Variational Quantum Classifier (VQC), a hybrid quantum-classical algorithm built from two parameterized circuits: a feature map that encodes classical data into qubit rotations (the Pauli or ZZ feature map, both angle-embedding circuits) and an ansatz whose parameters are trained classically (Real Amplitudes or EfficientSU2). The number of qubits equals the number of features, and the circuits are repeated twice with linear, circular, or full CNOT entanglement. The VQC carries the argument because every reported comparison—feature count, feature-selection method, training size, encoding, ansatz, and entanglement—is a variation on this circuit recipe.","core_discovery":"The paper's central claim is that the Variational Quantum Classifier, as implemented in Qiskit's machine learning library, is a usable and reasonably accurate tool for pulsar candidate classification on the HTRU-2 dataset. Concretely, a configuration with three features selected by a statistical score or by absolute correlation with the class, a ZZ or Pauli feature map, and an EfficientSU2 ansatz achieves accuracy up to 0.950 and MCC up to 0.670. The paper attributes the result to the interaction of circuit components rather than to any single ingredient: the best-performing models all use an EfficientSU2 ansatz, while using the data-encoding circuit itself as the ansatz gives high precision but much lower accuracy. The experiments are presented as evidence that hybrid quantum-classical classifiers can be applied to real astronomical surveys now, with tests on real quantum hardware and noisy simulators left to future work.","pith_inferences":["The paper does not compare the VQC with classical baselines on the same split, so the practical inference to draw is feasibility rather than superiority; a direct classical benchmark would put the 0.950 number in context.","If the feature-selection order is ambiguous, the result can be stress-tested by repeating the three-feature configuration with feature selection nested inside every training fold.","The strong dependence on entanglement pattern suggests that entanglement topology is a free hyperparameter of quantum classifiers, analogous to the kernel choice in classical support-vector machines, and worth tuning systematically.","Because the best numbers come from a search over many configurations, rerunning the same experiment with a pre-registered configuration list would show how much of the 0.950 is attributable to the search itself."],"forward_implications":["A three-qubit VQC is enough to reach the paper's best results, so the practical resource cost of the classifier stays low on near-term hardware.","Circuit identity matters: every top configuration uses the EfficientSU2 ansatz, while reusing the feature map as the ansatz changes the precision-recall tradeoff and lowers accuracy.","Feature-selection method shifts the error balance: FS1 gives better recall and MCC, while FS2 gives slightly better precision at similar accuracy.","Training on 180 samples produced higher accuracy than training on 300 samples in this setup, which the paper reads as a sign that the model's performance depends on batch composition.","The results support treating hybrid quantum-classical classification as a candidate screening tool for pulsar surveys, with real-device and noisy-simulation tests as the next step."],"supporting_citations":[{"why":"Supplies the HTRU-2 dataset, its eight features, and the class labels that define the pulsar classification task.","marker":"[31]"},{"why":"Presents the VQC algorithm, the hybrid quantum-classical classifier under test.","marker":"[62]"},{"why":"Surveys VQC applications and frames the algorithm's recent use.","marker":"[63]"},{"why":"Provides the angle-embedding feature-map construction behind the Pauli and ZZ feature maps.","marker":"[72]"},{"why":"Supports the use of the Matthews correlation coefficient for binary classification on imbalanced data.","marker":"[42]"},{"why":"Supports the choice of MCC over ROC when the dataset is imbalanced.","marker":"[43]"},{"why":"Establishes the feature-selection rationale used to reduce the eight HTRU-2 features.","marker":"[46]"}],"fun_headline_variants":["Qiskit VQC hits 95% accuracy on pulsar classification","Quantum classifier reaches 0.950 accuracy on pulsar candidates","Pulsar sorting via Qiskit VQC: 95% accuracy","Variational quantum classifier: 95% accuracy for pulsar detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 0.950 accuracy is only a trustworthy estimate of future performance if feature selection was fitted on the training portion alone and if the best configurations were not chosen by looking at the test set; the paper does not state exactly when the feature-selection step was applied.","fun_headline_variants_meta":{"raw":{"variants":["Qiskit VQC hits 95% accuracy on pulsar classification","Quantum classifier reaches 0.950 accuracy on pulsar candidates","Pulsar sorting via Qiskit VQC: 95% accuracy","Variational quantum classifier: 95% accuracy for pulsar detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2656,"prompt_tokens":824,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1755}},"tokens_in":440,"tokens_out":1832,"duration_ms":11123,"temperature":1.0,"reasoning_tokens":1755,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:13:25.545104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the three-feature VQC configuration with feature selection fitted inside each training fold of a cross-validation loop on the full HTRU-2 dataset, and compare the held-out accuracy with the reported 0.950. If the cross-validated accuracy falls materially below 0.950, the headline number came from leakage or from selecting the best of many configurations on the test set.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HTRU-2 dataset, its eight features, and the class labels that define the pulsar classification task."},{"cited_title":"Saxena and A","cited_arxiv_id":null,"evidence_quote":"Presents the VQC algorithm, the hybrid quantum-classical classifier under test."},{"cited_title":"Maheshwari, D","cited_arxiv_id":null,"evidence_quote":"Surveys VQC applications and frames the algorithm's recent use."},{"cited_title":"Suzuki, H","cited_arxiv_id":null,"evidence_quote":"Provides the angle-embedding feature-map construction behind the Pauli and ZZ feature maps."},{"cited_title":"Chicco and G","cited_arxiv_id":null,"evidence_quote":"Supports the use of the Matthews correlation coefficient for binary classification on imbalanced data."},{"cited_title":"Chicco and G","cited_arxiv_id":null,"evidence_quote":"Supports the choice of MCC over ROC when the dataset is imbalanced."},{"cited_title":"Kohavi and G","cited_arxiv_id":null,"evidence_quote":"Establishes the feature-selection rationale used to reduce the eight HTRU-2 features."}],"review_version":1}