{"id":"591ca87e-2605-4b01-a458-55d3f84c4c42","arxiv_id":"2508.13932","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of 53 automated feature engineering methods reports they are hard to use, poorly documented, and lack time and memory constraints, pointing to usability as an unmet need.","lead":"This preprint surveys 53 automated feature engineering tools for tabular machine learning and finds they are generally hard to use, under-documented, and lack active communities. It argues that none of these tools let users set time or memory limits, a gap the authors see as essential for practical automation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Universal negative on resource constraints is load-bearing; the 53-method sample's completeness is unverified in the abstract.","rationale":"The paper's strongest claim is a universal negative about all AutoFE methods, which is empirically strong and requires exhaustive or defensibly representative sampling. The reader's weakest assumption identifies exactly this: the sample completeness and the appropriateness of the criteria. I agree. Since the abstract gives no methodology, the correct verdict remains UNVERDICTED: not enough evidence to confirm or reject. There is no reason to move to ACCEPT, REJECT, or CONDITIONAL based on the abstract alone. The proposed concrete test would settle the key question: if the full paper's list omits a method with resource constraints, the headline claim fails; if the list is complete and definitions are precise, the claim could stand. This is the single most load-bearing concern because every other stated finding ('hard to use', 'lack documentation', 'no active communities') is weaker and more context-dependent, but the 'no method allows time/memory constraints' is crisp and falsifiable. The stress-test pass confirms the reader's verdict and identifies a specific, actionable check.","tokens_in":653,"tokens_out":1875,"duration_ms":21716,"concrete_test":"Compile a list of AutoFE methods from existing surveys (e.g., 'Automated Feature Engineering: A Survey', recent AutoML benchmarks) and compare it to the 53 methods in the full paper. For every method in the union, inspect its official repository/API for any parameter that constrains time or memory use (e.g., 'time_limit', 'max_memory', 'max_runtime', or a timeout argument). If any method inside or outside the 53 provides such a constraint, the abstract's universal negative is false. Additionally, take five methods from the full paper's evaluation and have two independent raters apply the paper's usability criteria; if inter-rater agreement is low, the 'hard to use' claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central conclusion is a universal negative: 'no method allows users to set time and memory constraints.' To support this, the survey must establish that the 53 methods surveyed are either exhaustive or a representative sample of the entire AutoFE landscape, and it must define 'resource constraints' operationally (e.g., does a runtime limit parameter count, or must it include memory caps? Does an indirect early-stopping mechanism count?). The abstract provides neither a sampling methodology nor a definition of criteria, so a single counterexample—a method outside the 53, or a method inside the 53 misclassified because the paper's definition of 'allows time constraints' is narrower than what the API exposes—would falsify the headline claim. Similarly, 'hard to use, lack documentation, and have no active communities' are subjective judgments; without a reproducible scoring protocol (e.g., measuring documentation length, README completeness, GitHub activity over a fixed window, or user studies), these assessments are not verifiable. Because only the abstract is available, there is no way to check whether the full paper provides such evidence. This is a load-bearing concern because the paper's novelty rests on these negative empirical findings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a survey of 53 automated feature engineering (AutoFE) methods for tabular data, focusing on usability for practitioners. Based on the abstract, the authors claim that the surveyed methods are generally hard to use, poorly documented, and lack active communities, and that no method allows users to set time and memory constraints. The abstract frames these findings as evidence that future work should emphasize usable, well-engineered AutoFE methods.","tokens_in":932,"tokens_out":1556,"duration_ms":18080,"significance":"If the findings are valid, the paper addresses a genuinely important gap: practitioners need tools that are not only accurate but also usable, documented, and resource-aware. The claimed universal negative about resource constraints, in particular, would be a strong and actionable signal for the AutoFE community. The paper also has the merit of formulating clear, falsifiable empirical claims. However, the abstract alone provides no methodology, data, or scoring protocol, so the significance can be assessed only conditionally. The study would be valuable if the full paper substantiates the survey protocol and the operational definitions behind its judgments.","major_comments":[{"comment":"The central claim that 'no method allows users to set time and memory constraints' is a universal negative over all AutoFE methods. The abstract does not define what counts as allowing such constraints (e.g., does a runtime limit parameter count, or must there be an explicit memory cap? Does an early-stopping mechanism qualify?). Without an operational definition and without evidence that the 53 surveyed methods are a representative or exhaustive sample of the AutoFE landscape, a single counterexample would falsify the headline claim. This is load-bearing because the paper's novelty rests on this negative empirical finding.","section":"Abstract"},{"comment":"The judgments that methods are 'hard to use, lack documentation, and have no active communities' are presented as objective findings, but the abstract gives no reproducible scoring protocol. For example, is documentation assessed by length, completeness, or example availability? Is community activity measured by commit frequency, issue response time, or contributor count over a fixed window? Without such a protocol, these assessments are not verifiable, and the general conclusion of 'hard to use' is not established.","section":"Abstract"},{"comment":"The sampling methodology is entirely absent. The abstract does not state how the 53 methods were selected, what inclusion/exclusion criteria were used, or how completeness with respect to the AutoFE literature was ensured. Since the survey claims to be the first investigation of usability in this area, the completeness and representativeness of the sample are essential. The abstract's unsupported sample description undermines the strength of both the universal negative and the qualitative usability conclusions.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'which we see as a necessity for usable automation' is a normative framing rather than an empirical finding. The paper should distinguish the authors' design stance from the survey data, and ideally justify this stance with prior work or practitioner interviews.","section":"Abstract"},{"comment":"The abstract states that AutoFE methods 'have never been investigated regarding their usability for practitioners.' This is a strong claim about the literature. A brief citation or a systematic search statement would help substantiate it.","section":"Abstract"},{"comment":"The term 'AutoFE' is used without defining it; while common in the field, a full name at first use would improve accessibility for readers from adjacent areas.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based only on the abstract because the full text was not available. The central empirical claims are potentially significant, but the abstract provides no evidence. If the full paper contains a rigorous survey protocol, a clear operational definition of resource constraints, and a reproducible scoring rubric, the paper could be acceptable. If those elements are missing, the claims would need substantial revision. My 'uncertain' verdict reflects the inability to verify the load-bearing methodology from the provided material, not a detected flaw in the abstract's reasoning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a survey that claims to be the first usability-focused audit of 53 AutoFE methods, and the headline finding—that the methods are generally hard to use, under-documented, have no active communities, and no support for time/memory constraints—is exactly the kind of concrete, actionable contribution the AutoML community could use. But the abstract alone cannot support the universal negative, and your referee should demand to see the sampling and scoring before buying it.\n\nWhat the paper does well: it identifies a genuine gap. Feature engineering is still a bottleneck in practical tabular ML, and most AutoFE papers focus on accuracy gains, not on whether a practitioner can actually run the tool. Surveying 53 methods is a nontrivial amount of work. The abstract is clear about the finding and about the normative stance ('which we see as a necessity'); that transparency is good.\n\nThe soft spots are real, and they sit exactly where the stress-test note points. The central claim is that no method allows users to set time and memory constraints. That is a universal negative over a 53-method sample, so the paper needs to show the sample is either complete or representative of the AutoFE ecosystem, and it needs an operational definition of 'allows.' Does a hard runtime limit count? Does an epoch-based early stopping count? What about indirectly setting memory via batch size? If the rubric is narrower than what APIs expose, a counterexample could flip the headline. Similarly, 'hard to use' and 'lack documentation' are subjective unless there is a reproducible scoring protocol—for example, a rubric for README length, API completeness, or a user study. Without that, the findings read as impressions, not evidence. I can’t tell from the abstract whether the full paper provides these; if it doesn’t, the survey is a useful mapping but not a rigorous one.\n\nMy take: this deserves a serious referee. The topic matters, the scale is impressive, and the conclusion, if verified, would be a good push for better engineering in AutoFE. I’d send it out with reviewers who know the method landscape and who will press on sampling and scoring. If the full text is as careful as the abstract is clear, it’s publishable. If not, it’s a workshop-level takeaway. Either way, it earns a look.","headline":"A praxis-first AutoFE survey with an actionable verdict, but the universal negative on resource constraints needs more than an abstract to be trusted.","tokens_in":1340,"tokens_out":2521,"would_cite":false,"duration_ms":24736,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing automated feature engineering methods are generally hard to use, poorly documented, and lack active communities; none of 53 surveyed tools lets users set time or memory constraints.","keywords":["automated feature engineering","tabular data","usability","practitioner survey","software documentation","community activity","resource constraints","machine learning"],"falsifier":"Locate any automated feature engineering method, among the 53 or in the wider literature, whose user interface or documentation exposes settable time or memory limits; a repository scan for flags like 'timeout' or 'max_memory' would provide the evidence. Even one such method would falsify the universal negative.","tokens_in":619,"feed_emoji":"⚙️","tokens_out":2505,"duration_ms":23660,"temperature":0.7,"pith_summary":"This paper surveys 53 automated feature engineering (AutoFE) methods and asks a question that has been left unexamined: can practitioners actually use these tools? It argues that most methods are hard to use, have sparse documentation, and attract little community activity. Most importantly, it claims that no surveyed method allows users to set time or memory constraints, which the authors consider necessary for usable automation. If true, the AutoFE field has been optimizing for predictive gains while neglecting the basic engineering needs of practitioners.","feed_headline":"No automated feature engineering tool respects time or memory limits","feed_subtitle":"A survey of 53 tools finds poor docs, inactive communities, and no way to cap compute.","key_machinery":"The key machinery is the evaluative framework itself: a set of usability criteria — ease of use, documentation quality, community activity, and availability of time/memory constraints — applied uniformly across 53 automated feature engineering methods. This framework transforms 'usability' from a vague concern into a checkable property, and it is what produces the universal negative about resource constraints.","core_discovery":"The paper's central claim is that usability has never been a design criterion in automated feature engineering research. By examining 53 methods, the authors find that, in general, these methods are hard to use, lack documentation, and have no active communities. They further report a universal negative: no method allows users to set time and memory constraints, even though such constraints are a necessity for practical automation. The survey concludes by calling for future work on usable, well-engineered AutoFE methods.","pith_inferences":["The authors stop short of saying so, but the absence of resource limits may be a larger adoption barrier than poor documentation: without time or memory caps, practitioners cannot safely hand over training jobs that might run for days or exhaust available memory.","A testable follow-up would be to add time and memory constraints to an existing AutoFE method and measure whether adoption or user satisfaction changes; the survey's criteria suggest such a wrapper would set a tool apart from the entire current field.","If the same usability criteria were applied to end-to-end AutoML frameworks that hide feature engineering behind an interface, many might also fail on documentation and resource controls even though practitioners use them successfully, implying the survey's negative verdict might not generalize to the broader automation ecosystem."],"forward_implications":["If the survey is correct, practitioners cannot rely on current AutoFE methods in production settings, since no method lets them cap runtime or memory usage.","Future AutoFE research would need to treat usability and resource management as first-class design goals, not optional add-ons.","Documentation and community health would become standard evaluation criteria alongside predictive performance in AutoFE benchmarks.","A method that adds explicit time and memory limits could immediately distinguish itself from the rest of the surveyed field."],"supporting_citations":[],"fun_headline_variants":["AutoFE tools fail usability: no time or memory limits in 53 methods","53 AutoFE tools: hard to use, poorly documented, no compute caps","Automated feature engineering is unusable: survey finds no time/memory limits","Feature engineering automation: 53 tools, zero usability or compute limits"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusion that no method allows resource constraints depends on the 53 surveyed methods being a representative slice of the AutoFE landscape, and on documentation, community activity, and resource limits being the right list of usability needs.","fun_headline_variants_meta":{"raw":{"variants":["AutoFE tools fail usability: no time or memory limits in 53 methods","53 AutoFE tools: hard to use, poorly documented, no compute caps","Automated feature engineering is unusable: survey finds no time/memory limits","Feature engineering automation: 53 tools, zero usability or compute limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":929,"prompt_tokens":617,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":361,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":361,"tokens_out":312,"duration_ms":3768,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:48:30.381884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Locate any automated feature engineering method, among the 53 or in the wider literature, whose user interface or documentation exposes settable time or memory limits; a repository scan for flags like 'timeout' or 'max_memory' would provide the evidence. Even one such method would falsify the universal negative.","supporting_citations":[],"review_version":1}