REVIEW 2 major objections 42 references
An online platform recommends complete machine learning pipelines to non-experts by reasoning over encoded expert knowledge.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-27 20:12 UTC pith:NLWEP7DZ
load-bearing objection This is a high-level design sketch for an FOL-driven ML pipeline recommender with no rules, examples, or implementation shown. the 2 major comments →
Public Machine Learning Solver Framework for Novices in the Machine Learning Domain
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors describe a semi-automated platform that combines expert cheat sheets and decision-support criteria to recommend complete machine learning solution pipelines. The platform automatically extracts dataset characteristics, incorporates transfer learning, and employs first-order logic to reason over a knowledge base of algorithms and selection rules, producing ranked recommendations through a user-friendly interface that can be extended incrementally and updated via expert crowdsourcing.
What carries the argument
First-order logic reasoning over an expert knowledge base of algorithms and selection criteria, paired with automatic data feature extraction and transfer learning to generate ranked pipeline recommendations.
Load-bearing premise
Expert knowledge about suitable algorithms and pipelines can be captured in first-order logic rules that, together with extracted data features, produce recommendations appropriate for non-experts.
What would settle it
A controlled comparison in which the platform and existing AutoML tools each generate pipelines for the same set of real user datasets, followed by expert evaluation of recommendation suitability and downstream model performance.
If this is right
- Non-experts receive ranked lists of complete pipelines rather than isolated algorithm suggestions.
- Recommendations incorporate multiple criteria such as accuracy, transparency, and data requirements at once.
- The knowledge base can be extended with new algorithms and domain rules without rebuilding the system.
- Continuous updates occur through an integrated crowdsourcing platform for machine learning experts.
- The platform remains publicly accessible and free while maintaining transparency through explicit logical rules.
Where Pith is reading between the lines
- Domain experts from other fields could apply machine learning more consistently if the recommendations prove reliable on their data types.
- The explicit logical encoding might allow users to inspect and understand the reasons behind each pipeline suggestion.
- Similar rule-based guidance structures could be developed for other technical domains that currently require specialized training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a Public Machine Learning Solver Framework that integrates expert cheat sheets and decision-support criteria into a semi-automated online platform. It uses first-order logic reasoning over a crowdsourced knowledge base, automatic extraction of dataset characteristics (class imbalance, missing values, etc.), and transfer learning to recommend complete ranked ML pipelines to non-experts, claiming to be the first free public system that systematically operationalizes expert knowledge in this manner.
Significance. If the described architecture were implemented with concrete, validated rules and shown to produce suitable recommendations superior to existing AutoML or cheat-sheet approaches, the work could offer a transparent, incrementally updatable alternative for novices. The current manuscript, however, supplies only an architectural outline with no rule set, no feature-extractor implementation, and no reasoning trace, so the claimed significance remains unrealized.
major comments (2)
- [Abstract / Platform Architecture] Abstract and platform-architecture section: the central claim that the system 'operationalizes expert knowledge' via first-order logic rules combined with automatic feature extraction and transfer learning is unsupported, as the manuscript contains neither the rule set, pseudocode for the extractors, nor any worked example showing that extracted features plus logic produce ranked pipelines on even one sample dataset.
- [Abstract] The assertion that the platform 'suggests a complete pipeline tailored to the user's problem' and is 'superior to existing approaches' rests entirely on the untested assumption that the FOL reasoner will yield suitable outputs; without any implementation or comparison data this assertion cannot be evaluated.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive comments. The manuscript presents a conceptual proposal for a public ML solver framework rather than a fully implemented and evaluated system. We address each major comment below and will revise the manuscript to clarify the scope and add illustrative details.
read point-by-point responses
-
Referee: [Abstract / Platform Architecture] Abstract and platform-architecture section: the central claim that the system 'operationalizes expert knowledge' via first-order logic rules combined with automatic feature extraction and transfer learning is unsupported, as the manuscript contains neither the rule set, pseudocode for the extractors, nor any worked example showing that extracted features plus logic produce ranked pipelines on even one sample dataset.
Authors: We agree that the manuscript supplies only a high-level architectural outline without concrete rule sets, extractor pseudocode, or worked examples. This reflects the paper's focus on the overall design and crowdsourced extensibility rather than a complete implementation. In revision we will add pseudocode for the data-characteristic extractors and a concrete worked example (including extracted features, sample FOL rules, and resulting ranked pipeline) on one public dataset to illustrate the reasoning process. revision: yes
-
Referee: [Abstract] The assertion that the platform 'suggests a complete pipeline tailored to the user's problem' and is 'superior to existing approaches' rests entirely on the untested assumption that the FOL reasoner will yield suitable outputs; without any implementation or comparison data this assertion cannot be evaluated.
Authors: The current wording in the abstract presents the pipeline suggestion and potential superiority as design objectives of the proposed framework. We acknowledge that these remain untested without implementation or benchmarks. We will revise the abstract and architecture section to state these as intended outcomes of the architecture, to be validated once the knowledge base and reasoner are populated, rather than as demonstrated results. revision: yes
Circularity Check
High-level system design proposal with no derivations or predictions
full rationale
The manuscript is an architectural proposal for an FOL-based recommender platform. It outlines categories of existing systems, describes desired features (feature extraction, transfer learning, crowdsourcing KB updates), and asserts novelty, but supplies no equations, no fitted parameters, no predictions, and no derivation chain. The central claim is a 'first such framework' assertion, not a result derived from inputs. No self-citations, ansatzes, or reductions to fitted data appear. This is a standard non-circular design sketch.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Expert knowledge for algorithm and pipeline selection can be represented as first-order logic rules over data characteristics and criteria such as accuracy and transparency.
invented entities (1)
-
Public ML Solver Framework platform
no independent evidence
read the original abstract
Solving machine learning problems is complex and typically reserved for experts. Over the past two decades, systems have emerged to support non-experts. Based on our review, we identify three categories: (1) fully automated AutoML systems, (2) expert cheat sheets for algorithm selection, and (3) decision-support systems using selection criteria (accuracy, transparency, data requirements). We propose a new platform combining categories 2 and 3 to deliver semi-automated, intelligent solution recommendations for non-experts. Unlike existing approaches that recommend a single algorithm, our platform suggests a complete pipeline tailored to the user's problem. It integrates expert-defined selection criteria with transfer learning and automatically extracts data characteristics (e.g., class imbalance, missing values) from user-provided datasets. The platform uses first-order logic to reason over its knowledge base and recommends suitable algorithms ranked by relevance. It features a user-friendly interface and connects to a crowdsourcing platform for ML experts, ensuring continuous updates. The platform is built incrementally, allowing seamless integration of new algorithms, criteria, and domain knowledge. To our knowledge, this is the first free, publicly accessible online framework that systematically captures and operationalizes expert knowledge to guide non-experts in solving ML problems in a structured, transparent manner.
Figures
Reference graph
Works this paper leans on
-
[1]
D-smartml: A distributed automated machine learning framework
Ahmed Abd Elrahman, Mohamed El Helw, Radwa Elshawi, and Sherif Sakr. D-smartml: A distributed automated machine learning framework. In2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 1215–1218. IEEE
-
[2]
A new look at the statistical model identification.IEEE transactions on automatic control, 19(6):716–723, 1974
Hirotugu Akaike. A new look at the statistical model identification.IEEE transactions on automatic control, 19(6):716–723, 1974
1974
-
[4]
A roadmap of clustering algorithms: finding a match for a biomedical application.Briefings in Bioinfor- matics, 10(3):297–314, 2009
Bill Andreopoulos, Aijun An, Xiaogang Wang, and Michael Schroeder. A roadmap of clustering algorithms: finding a match for a biomedical application.Briefings in Bioinfor- matics, 10(3):297–314, 2009
2009
-
[5]
A survey on machine-learning techniques in cognitive radios.IEEE Communications Surveys & Tutorials, 15(3):1136– 1159, 2012
Mario Bkassiny, Yang Li, and Sudharman K Jayaweera. A survey on machine-learning techniques in cognitive radios.IEEE Communications Surveys & Tutorials, 15(3):1136– 1159, 2012. 14
2012
-
[6]
Au- tostacker: A compositional evolutionary learning system
Boyuan Chen, Harvey Wu, Warren Mo, Ishanu Chattopadhyay, and Hod Lipson. Au- tostacker: A compositional evolutionary learning system. InProceedings of the Genetic and Evolutionary Computation Conference, pages 402–409
-
[7]
Au- tostacker: A compositional evolutionary learning system
Boyuan Chen, Harvey Wu, Warren Mo, Ishanu Chattopadhyay, and Hod Lipson. Au- tostacker: A compositional evolutionary learning system. InProceedings of the Genetic and Evolutionary Computation Conference, pages 402–409, 2018
2018
-
[8]
A survey on machine learning: concept, algo- rithms and applications.International Journal of Innovative Research in Computer and Communication Engineering, 5(2):1301–1309, 2017
Kajaree Das and Behera Rabi Narayan. A survey on machine learning: concept, algo- rithms and applications.International Journal of Innovative Research in Computer and Communication Engineering, 5(2):1301–1309, 2017
2017
-
[9]
A comparison of multiple classification methods for diagnosis of parkinson disease.Expert Systems with Applications, 37(2):1568–1572, 2010
Resul Das. A comparison of multiple classification methods for diagnosis of parkinson disease.Expert Systems with Applications, 37(2):1568–1572, 2010
2010
-
[10]
Recipe: a grammar-based framework for automatically evolving classification pipelines
Alex GC de S´ a, Walter Jos´ e GS Pinto, Luiz Otavio VB Oliveira, and Gisele L Pappa. Recipe: a grammar-based framework for automatically evolving classification pipelines. In European Conference on Genetic Programming, pages 246–261. Springer
-
[12]
AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020
work page internal anchor Pith review Pith/arXiv arXiv 2003
-
[14]
Survey of machine learning algorithms for disease diagnostic.Journal of Intelligent Learning Systems and Applications, 9(01):1, 2017
Meherwar Fatima and Maruf Pasha. Survey of machine learning algorithms for disease diagnostic.Journal of Intelligent Learning Systems and Applications, 9(01):1, 2017
2017
-
[15]
Multi-interval discretization of continuous-valued attributes for classification learning
Usama M Fayyad and Keki B Irani. Multi-interval discretization of continuous-valued attributes for classification learning. InIjcai, volume 93, pages 1022–1029. Citeseer, 1993
1993
-
[16]
Practical automated machine learning for the automl challenge 2018
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Practical automated machine learning for the automl challenge 2018. InInterna- tional Workshop on Automatic Machine Learning at ICML, pages 1189–1232
2018
-
[17]
Practical automated machine learning for the automl challenge 2018
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Practical automated machine learning for the automl challenge 2018. InInterna- tional Workshop on Automatic Machine Learning at ICML, pages 1189–1232, 2018
2018
-
[18]
Auto-sklearn: efficient and robust automated machine learning
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Auto-sklearn: efficient and robust automated machine learning. Automated Machine Learning, pages 113–134, 2018
2018
-
[19]
P4ml: A phased performance-based pipeline planner for automated machine learning
Yolanda Gil, Ke-Thia Yao, Varun Ratnakar, Daniel Garijo, Greg Ver Steeg, Pedro Szekely, Rob Brekelmans, Mayank Kejriwal, Fanghao Luo, and I-Hui Huang. P4ml: A phased performance-based pipeline planner for automated machine learning. InAutoML Workshop at ICML
-
[20]
Pspso: A package for parameters selection using particle swarm optimization.SoftwareX, 15:100706, 2021
Ali Haidar, Matthew Field, Jonathan Sykes, Martin Carolan, and Lois Holloway. Pspso: A package for parameters selection using particle swarm optimization.SoftwareX, 15:100706, 2021. 15
2021
-
[21]
The determination of the order of an autoregression
Edward J Hannan and Barry G Quinn. The determination of the order of an autoregression. Journal of the Royal Statistical Society: Series B (Methodological), 41(2):190–195, 1979
1979
-
[22]
Supervised machine learning: A re- view of classification techniques.Emerging artificial intelligence applications in computer engineering, 160(1):3–24, 2007
Sotiris B Kotsiantis, I Zaharakis, and P Pintelas. Supervised machine learning: A re- view of classification techniques.Emerging artificial intelligence applications in computer engineering, 160(1):3–24, 2007
2007
-
[23]
Auto-WEKA: Automatic model selection and hyperparameter optimization in WEKA, pages 81–95
Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin Leyton-Brown. Auto-WEKA: Automatic model selection and hyperparameter optimization in WEKA, pages 81–95. Springer, Cham, 2019
2019
-
[24]
H2o automl: Scalable automatic machine learning
Erin LeDell and Sebastien Poirier. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020
2020
-
[25]
Smartml: A meta learning-based framework for auto- mated selection and hyperparameter tuning for machine learning algorithms
Mohamed Maher and Sherif Sakr. Smartml: A meta learning-based framework for auto- mated selection and hyperparameter tuning for machine learning algorithms. InEDBT: 22nd International Conference on Extending Database Technology
-
[26]
Metwalli
Sara A. Metwalli. How to choose the right machine learning al- gorithm for your application.https://towardsdatascience.com/ how-to-choose-the-right-machine-learning-algorithm-for-your-application-1e36c32400b9,
-
[27]
Consult´ e le 6 juin 2026
2026
-
[28]
Machine learning algorithm cheat sheet for azure machine learning designer
Microsoft. Machine learning algorithm cheat sheet for azure machine learning designer. 2020
2020
-
[29]
Ml-plan: Automated machine learning via hierarchical planning.Machine Learning, 107(8):1495–1515, 2018
Felix Mohr, Marcel Wever, and Eyke H¨ ullermeier. Ml-plan: Automated machine learning via hierarchical planning.Machine Learning, 107(8):1495–1515, 2018
2018
-
[31]
In- ternational journal of innovative research in computer and communication engineering
Ms Gaurangi Patil, Ms Varsha Galande, Mr Vedant Kekan, and Ms Kalpana Dange. In- ternational journal of innovative research in computer and communication engineering. Sentiment analysis using support vector machine, 2(1):2607–2612, 2014
2014
-
[32]
DarwinML: A Graph-based Evolutionary Algorithm for Automated Machine Learning
Fei Qi, Zhaohui Xia, Gaoyang Tang, Hang Yang, Yu Song, Guangrui Qian, Xiong An, Chunhuan Lin, and Guangming Shi. Darwinml: A graph-based evolutionary algorithm for automated machine learning.arXiv preprint arXiv:1901.08013, 2018
work page internal anchor Pith review Pith/arXiv arXiv 1901
-
[33]
Automated machine learn- ing with monte-carlo tree search.arXiv preprint arXiv:1906.00170, 2019
Herilalaina Rakotoarison, Marc Schoenauer, and Mich` ele Sebag. Automated machine learn- ing with monte-carlo tree search.arXiv preprint arXiv:1906.00170, 2019
-
[34]
A quick review of machine learning algorithms
Susmita Ray. A quick review of machine learning algorithms. In2019 International con- ference on machine learning, big data, cloud and parallel computing (COMITCon), pages 35–39. IEEE
-
[35]
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019
work page internal anchor Pith review Pith/arXiv arXiv 1908
-
[36]
How to select a suitable machine learning algorithm: A feature-based, scope-oriented selection framework
R Sala, M Zambetti, F Pirola, and R Pinto. How to select a suitable machine learning algorithm: A feature-based, scope-oriented selection framework. In23rd Summer School” Francesco Turco”-Industrial Systems Engineering 2018, volume 2018, pages 87–93. AIDI- Italian Association of Industrial Operations Professors. 16
2018
-
[38]
A review of clustering techniques and developments.Neurocomputing, 267:664–681, 2017
Amit Saxena, Mukesh Prasad, Akshansh Gupta, Neha Bharill, Om Prakash Patel, Aruna Tiwari, Meng Joo Er, Weiping Ding, and Chin-Teng Lin. A review of clustering techniques and developments.Neurocomputing, 267:664–681, 2017
2017
-
[39]
Estimating the dimension of a model.The annals of statistics, 6(2):461– 464, 1978
Gideon Schwarz. Estimating the dimension of a model.The annals of statistics, 6(2):461– 464, 1978
1978
-
[40]
Machine learning map.https://scikit-learn.org/stable/ tutorial/machine_learning_map/index.html, 2020
scikit-learn developers. Machine learning map.https://scikit-learn.org/stable/ tutorial/machine_learning_map/index.html, 2020. Consult´ e le 6 juin 2026
2020
-
[41]
Zoomed ranking: Selection of classification algorithms based on relevant performance information
Carlos Soares and Pavel B Brazdil. Zoomed ranking: Selection of classification algorithms based on relevant performance information. InEuropean conference on principles of data mining and knowledge discovery, pages 126–135. Springer
-
[42]
Yoel S´ anchez Berm´ udez.How to select the right machine learning approach?Degree project, 2013
2013
-
[43]
Guidelines to select machine learning scheme for classification of biomedical datasets
Ajay Kumar Tanwani, Jamal Afridi, M Zubair Shafiq, and Muddassar Farooq. Guidelines to select machine learning scheme for classification of biomedical datasets. InEuropean Conference on Evolutionary Computation, Machine Learning and Data Mining in Bioin- formatics, pages 128–139. Springer
-
[44]
Auto-weka: Combined selection and hyperparameter optimization of classification algorithms
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. InPro- ceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 847–855
-
[45]
Automated machine learning in practice: state of the art and recent results
Lukas Tuggener, Mohammadreza Amirian, Katharina Rombach, Stefan L¨ orwald, Anastasia Varlet, Christian Westermann, and Thilo Stadelmann. Automated machine learning in practice: state of the art and recent results. In2019 6th Swiss Conference on Data Science (SDS), pages 31–36. IEEE
-
[46]
Hyperparameter importance across datasets
Jan N Van Rijn and Frank Hutter. Hyperparameter importance across datasets. InPro- ceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2367–2376, 2018
2018
-
[47]
A novel evolutionary algorithm for automated machine learning focusing on classifier ensem- bles
Joao C Xavier-J´ unior, Alex A Freitas, Antonino Feitosa-Neto, and Teresa B Ludermir. A novel evolutionary algorithm for automated machine learning focusing on classifier ensem- bles. In2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pages 462–467. IEEE. 17
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.