{"id":"4ffed543-84ef-4426-8642-cfc2c5fc30d7","arxiv_id":"2412.06303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DSAI extracts interpretable latent features from text through five LLM stages and ranks them by a prominence score that measures how well each rule separates positive from negative examples.","lead":"This paper describes DSAI, a five-step AI pipeline that reads text datasets and writes short, human-readable rules describing what makes some examples stand out. It is meant for teams that want to understand large datasets without hiring domain experts to label features by hand.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSAI's data-grounding claim is not tested where it matters: unlike the direct baselines in §3.2, DSAI is never given flipped labels, so its 'superior grounding' could reflect label-chasing rather than data-driven adaptation.","rationale":"The paper makes a valuable empirical contribution: DSAI is a clearly described, cheap, multi-stage pipeline that recovers a substantial fraction of expert-defined criteria, and the recall results in Table 1 are a genuine positive. The direct-prompting experiments in §3.2 are also a thoughtful demonstration that LLMs can regurgitate priors rather than adapt to data. However, the central claim that DSAI is 'unbiased' and 'data-grounded' is not directly tested. The only behavioral test of data adaptation used in the paper—the flipped-label experiments—is applied to the direct baselines but not to DSAI itself. Since DSAI's prominence metric is computed from the same LLM-generated labels that define the positive/negative split, the pipeline could be selecting features that merely separate the LLM's own annotations, which is not the same as grounding in the underlying data. The reader's weakest assumption about label noise is related and partially overlaps with this concern: if the GPT-4o labels are biased, DSAI inherits that bias. But the more load-bearing gap is the absence of a flipped-label or label-permutation test on DSAI, because that test would directly arbitrate between 'grounded in data' and 'optimized against LLM-produced labels.' The manual DP validation in §5.2 is a useful check on the prominence metric's correlation with human judgments, but it samples only a small number of high-prominence features and does not address whether the features themselves would change if the labels changed. The abstract's description of 'synthetic datasets with known ground-truth features' is also inaccurate—the two evaluation datasets are real text datasets with expert-defined criteria—which further weakens the apparent strength of the grounding evidence. None of this shows the method is wrong; it shows the central claim is currently under-supported. The appropriate disposition remains CONDITIONAL, pending a direct label-flip test on DSAI, so the reader's verdict does not need to change.","tokens_in":22646,"tokens_out":3879,"duration_ms":39334,"concrete_test":"Run the full DSAI pipeline on the slogan and title datasets with the positive and negative labels swapped, and as a control with labels randomly permuted, keeping all prompts and thresholds fixed. If the high-prominence features do not invert or change materially—for example, if the top-20 features under flipped labels overlap strongly with the top-20 features under original labels—then DSAI is relying on prior knowledge or annotation bias rather than data grounding. A quantitative criterion: compute the Jaccard overlap of the top-20 features between the original and flipped-label runs; if overlap remains above 0.5 while the labels have been fully inverted, the data-grounding claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DSAI grounds features in data rather than in LLM prior knowledge. The evidence for this is indirect: recall of expert criteria (§5.1) and DP scores above 0.5 (Fig. 1). But the pipeline's prominence score, which drives feature selection, is computed as |2*P(positive|D_{p,l}) - 1| using the same GPT-4o-generated positive/negative labels described in Appendix A. Because the model both creates the labels and is then optimized against them, high prominence can be an artifact of the LLM's own annotation bias—the very bias the paper claims to remove. Direct baselines were tested for data adaptation by flipping labels (§3.2); DSAI is not. A truly data-grounded pipeline should invert its selected features when positives and negatives are swapped, or at least change them substantially. Without this test, the Figure 1 claim ('no DP score below 0.5') is not a demonstration of grounding; it is partly a selection artifact, and the limitation section (Sec. 8) concedes that model-based annotations remain a potential source of error. The manual DP validation in §5.2 covers only a small sample of high-prominence features and does not test whether the features themselves are data-driven.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DSAI, a five-stage LLM pipeline (perspective generation, perspective-value matching, value clustering, verbalization, and prominence-based selection) intended to extract interpretable latent features from text while avoiding reliance on LLM prior knowledge. The central claims are that the prominence intensity |2*P(positive|Dp,l)−1| quantifies each feature's discriminative power and that DSAI achieves superior data grounding relative to direct feature-generation prompting. Validation uses expert-defined criteria on slogan and research-title datasets, DP comparisons in Figure 1, a small manual DP check in Section 5.2, and qualitative real-world applications on news headlines, spam, and Reddit comments.","tokens_in":22907,"tokens_out":6919,"duration_ms":76028,"significance":"The paper addresses a real problem: LLM-generated features can be plausible yet ungrounded, and the Section 3.2 label-flip and no-data diagnostics for direct prompting are well-designed negative results. DSAI's source-to-feature traceability and adjustable prominence threshold are practically attractive, and the recall results against expert criteria are a genuine positive if they survive independent annotation. However, the central data-grounding claim is not currently established because the positive/negative labels are themselves LLM-produced (Appendix A), and the same labels appear to drive both the prominence-based selection and the DP evaluation. The paper would be significant if the pipeline were validated with independent labels and a label-flip test.","major_comments":[{"comment":"The DP validation is partly circular. Prominence is defined as |2*P(positive|Dp,l)−1|, and for positively verbalized features the DP score is P(positive|feature-present), where P(positive|Dp,l) is estimated from the same GPT-4o labels that also define the positive/negative split and are used to select features (Appendix A). For any feature that passes a positive prominence threshold τ>0, this quantity is at least (1+τ)/2>0.5 by construction. Thus the claim in Figure 1(a) that no DSAI feature has DP below 0.5 is not an independent test of grounding when DP is computed on the automatic labels. The authors should recompute DP using the independent manual annotations from Section 5.2 for all selected features, or at least report the manual-only DP distribution separately and specify which labels were used for each curve in Figure 1.","section":"§4 (#4–#5), §3.2(c), Figure 1"},{"comment":"The comparison with direct prompting is missing the key control. Section 3.2 tests whether POSDATA and MIXEDDATA adapt to flipped labels and shows that they do not; DSAI is never subjected to the same flipped-label test. Because Stages 4 and 5 derive directional scores and prominence from the provided labels, a flipped-label experiment is the minimal test of whether DSAI's selected features are driven by the data or merely follow whatever labels are supplied. Without this experiment, the conclusion that DSAI is more data-grounded than the direct baselines is unsupported, even though DSAI may produce different features under a label flip.","section":"§3.2(a) vs. §5"},{"comment":"The ground-truth labels themselves are not independent expert labels. Appendix A states that only 10–20 samples per dataset were manually annotated and that the GPT-4o prompts were optimized against these samples before annotating 3,000 data points. The high recall in Table 1 may therefore reflect alignment between the LLM annotator and the LLM feature extractor rather than recovery of genuinely independent expert criteria. The paper should report per-criterion agreement of the optimized GPT-4o annotations with the manual gold set, and should test robustness by rerunning at least the slogans and titles experiments with human labels or with a different annotator model. Section 8 concedes that model-based annotations can introduce bias, but the current manuscript does not quantify this risk.","section":"§3.1, Appendix A"},{"comment":"The manual DP validation is too limited to support the strong claim that prominence ranks features by their true discriminative power. Only 10 criteria per prominence bucket are sampled, no inter-annotator agreement is reported, and the paper does not state whether the manual feature-presence annotations were produced independently of the automatic labels used for prominence. The reported trend in Figure 1(b) is suggestive, but the sample size and the lack of independence information do not rule out the possibility that the trend reflects the LLM's own annotation consistency rather than data-level discriminative power.","section":"§5.2, Figure 1(b)"},{"comment":"Table 13, labeled as the Spam Detection Dataset, is identical to Table 11, which reports the Title dataset. As a result, the paper does not actually provide the claimed top/bottom prominence features for the spam dataset, and the qualitative statements about spam-specific patterns in Section 6 and Appendix G lack their supporting table. This is a factual reporting error that should be corrected.","section":"Appendix G, Table 13"}],"minor_comments":[{"comment":"The cross-reference to the NOCONTEXT results is unresolved (\"Appendix ??\"); the appendix containing those prompt outputs should be cited explicitly.","section":"§3.2(d)"},{"comment":"There is a typo in the criterion name \"Consise but not too simple\"; it should be \"Concise but not too simple.\"","section":"Table 3"},{"comment":"The text contains a typo, \"prominence fearures\" instead of \"prominence features,\" and the appendix would benefit from a proofread.","section":"Appendix G"},{"comment":"The footnote stating that the paper title was chosen by DSAI-generated criteria is a self-referential detail that does not contribute to the scientific content and may confuse readers; I recommend removing it.","section":"Introduction, footnote 1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major revision. The pipeline and the negative results for direct prompting are valuable, and the central defect is fixable with additional experiments: a flipped-label run of DSAI, independent human labels for the positive/negative groups, and a manual-only DP evaluation. I would not reject the paper outright, but I would not accept it without these controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper describes a clean, well-engineered pipeline for turning text collections into ranked, human-readable feature lists. The five stages—perspective generation, value matching, clustering, verbalization, prominence selection—are not new individually, but the integration with hidden task context is a real contribution. The recall results against expert-defined criteria are solid, and the cost analysis is transparent.\n\nWhat the paper does well: the recall of 100% for slogans and 83% for titles at high thresholds is genuine. The manual DP validation in Section 5.2 provides some independent signal, though small. The writing is clear, and the limitations section is honest about model-based annotation risks.\n\nThe main soft spot is the data-grounding claim. The stress-test note is right: DSAI is never given flipped labels. Direct baselines are tested for label adaptation; DSAI is not. Because prominence is |2*P(positive|D_{p,l}) - 1| and the positive/negative labels come from GPT-4o (optimized against 10–20 manual annotations), selecting features by prominence guarantees DP above 0.5 when DP is computed on those same labels. So Figure 1's \"no DP below 0.5\" is partly a selection artifact. The manual validation in Section 5.2 only samples a few high-prominence features, so it does not test whether the selection itself is data-driven.\n\nAlso, the abstract says \"synthetic datasets with known ground-truth features\" but the datasets are real text corpora with expert criteria. That is inaccurate. Another issue: code and prompts are promised but not provided, and the appendix has duplicated tables (Tables 11–13 appear identical in the manuscript). These are fixable.\n\nThe core approach is sound, and the prominence metric is useful when labels are externally defined (CTR, spam). The paper should add a flipped-label experiment for DSAI, release the prompts, and correct the abstract and duplicated tables.\n\nWho is this for: practitioners in data-centric AI and business analytics who want cheap, interpretable feature discovery. It deserves a serious referee, though the central claim needs strengthening. I would send it to review with a request for the flipped-label test and prompt release before acceptance.","headline":"A promising five-stage pipeline for interpretable feature extraction, but the 'data-grounding' claim needs a flipped-label test on DSAI itself.","tokens_in":23450,"tokens_out":2010,"would_cite":true,"duration_ms":20007,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs asked to extract dataset features mostly recite pre-trained knowledge; DSAI, a five-stage bottom-up pipeline, extracts expert-level features from the data itself and ranks them by a prominence score tied to true discriminative power.","keywords":["latent feature extraction","data grounding","large language models","interpretability","data-centric AI","prominence intensity","LLM bias","text data analysis"],"falsifier":"Run DSAI on a dataset whose true class structure contradicts GPT-4o's priors - for example, a slogan corpus where human experts rate clichéd, salesy slogans as 'good' and clever understated ones as 'bad' - and check whether the extracted features track the actual labels or drift toward the model's prior notion of good slogans; equivalently, feed DSAI the same labeled data with positive and negative labels swapped and see whether its features invert.","tokens_in":22429,"feed_emoji":"📊","tokens_out":15654,"duration_ms":123521,"temperature":0.7,"pith_summary":"This paper claims to have diagnosed a specific failure mode of LLMs as data analysts - they do not read the data, they recite what they already know - and to have built a pipeline that fixes it. The diagnosis is direct: when GPT-4o is asked to list features of 'high-quality' slogans or paper titles, it produces lists that overlap heavily with expert criteria even when given no data at all, and the lists barely change when input labels are flipped, evidence that the output is prior knowledge rather than evidence. The fix, DSAI, is a five-stage bottom-up pipeline that hides the domain from the model, generates perspectives from a small sample, assigns values to individual data points under each perspective, clusters those values, verbalizes each (perspective, label) pair as a positive criterion or an 'avoid' statement, and ranks features by prominence intensity, $|2P(positive|D_{p,l})-1|$, a rescaled measure of how much a feature concentrates in the positive class. On datasets with expert-defined ground truth, DSAI recovers most expert criteria, no DSAI feature has discriminative power below 0.5 whereas directly generated features frequently do, and higher prominence predicts higher discriminative power. If the claim is right, automated, interpretable, quantitatively ranked feature extraction becomes practical for data-centric AI, at roughly two and a half dollars per hundred sentences in the paper's accounting.","feed_headline":"Five-stage pipeline grounds LLM features in data, not priors","feed_subtitle":"Each extracted criterion gets a prominence score that tracks its real discriminative power.","key_machinery":"The central object is prominence intensity, the absolute value of the directional score $|2P(positive|D_{p,l})-1|$, where $D_{p,l}$ is the set of data points assigned cluster label $l$ under perspective $p$; the term inside is the fraction of positive examples in that cell, rescaled from $[0,1]$ to $[-1,1]$. The sign of the directional score decides whether the (perspective, label) pair becomes a direct criterion or an 'avoid' statement, and its magnitude is the feature's reliability ranking. This single number carries the argument because it converts an LLM's free-form criteria into a quantitative, user-tunable index: setting a prominence threshold trades coverage against discriminative power, and the paper validates that higher prominence predicts higher measured DP, with all DSAI features above chance. The second carrier is the pipeline's bottom-up structure, in which the task context is withheld and values are assigned per data point, so that every final criterion is an aggregation of observations rather than a recollection, which is what grounds the features in the data.","core_discovery":"On its own terms, the paper's discovery is that the data-grounding failure of LLM feature extraction is real, invisible, and fixable. It is real because the NODATA probe shows GPT-4o regenerates nearly all expert criteria from memory alone, and the flipped-label probes show direct generation is insensitive to the data: FLIPPED POSDATA yields features nearly identical to POSDATA, and FLIPPED MIXEDDATA closely matches MIXEDDATA. It is invisible because the regurgitated lists look expert-quality and overlap with ground truth, so a user would not notice the model was not reading the data; the only sign is that several directly generated features have DP below 0.5, meaning the model recommends features more common in the negative class. It is fixable through DSAI's design: the LLM is never told the domain or which class is 'high quality', perspectives are generated from a labeled sample, each data point is assigned values under every perspective, values are clustered, verbalized into criteria, and selected by prominence intensity, the absolute rescaled precision $|2P(positive|D_{p,l})-1|$ of each (perspective, label) cell. The validation reports DSAI matching direct methods on recall of expert criteria - all nine slogan criteria recovered at threshold 0.348, ten of twelve title criteria at threshold 0.692 - while every DSAI feature stays above DP 0.5, DP rises with prominence, and the pipeline's self-audits report 94%-plus consistency on stages #2, #3, and #4.","pith_inferences":["The paper's flipped-label probe could be pointed at DSAI itself: if its extracted features fail to invert when positive and negative labels are swapped, 'grounded in data' would actually mean 'grounded in the annotator model's beliefs,' and the grounding advantage over direct prompting would need reinterpreting.","Since all labels used for prominence and DP come from GPT-4o with prompts tuned on 10-20 annotations, a natural stress test is to re-label the same datasets with a different or weaker model and check whether DSAI's recall, DP, and grounding comparisons survive.","The prominence machinery extends beyond binary classification: scoring features by class-conditional or residual statistics would give the same ranking logic in multi-class and regression settings.","The NODATA probe suggests a cheap quantitative baseline the paper does not compute: the overlap between direct-prompt features and DSAI features could be reported as a 'prior-overlap' score, making the grounding claim directly measurable per feature."],"forward_implications":["Users can set a single prominence threshold to decide how many features to keep, and the validation shows higher thresholds reliably retain the most discriminative criteria while filtering out generic ones.","Because every extracted criterion is traceable to the data points behind it, the same pipeline can support interpretable classification, style transfer, and the automatic drafting of annotation guidelines without domain experts.","DSAI captures both dataset-wide traits and fine-grained subset-specific features - such as sarcastic undertone in Reddit comments - that direct LLM analysis tends to miss.","In a toy spam-classification test, the five highest-prominence criteria classify well while the five lowest-prominence criteria do not, indicating that the prominence ranking tracks practical utility.","Because the domain is never revealed to the model, the pipeline transfers to new domains - news headlines, spam, Reddit comments - without any domain-specific tuning, at a reported cost of about $2.44 per hundred sentences through ten perspectives."],"supporting_citations":[{"why":"Evidence that in-context learning fails to override an LLM's prior biases even with explicit prompts; this motivates the grounding problem DSAI targets.","marker":"Kossen et al., 2024"},{"why":"Quantifies how often LLMs follow internal priors when they conflict with provided context, the failure mode the paper probes with label flipping.","marker":"Wu et al., 2024"},{"why":"Survey cited for the limits of external-knowledge grounding, supporting the need for a bottom-up extraction pipeline.","marker":"Kenthapadi et al., 2024"},{"why":"Identifies GPT-4o, the model used for every pipeline stage and for the annotation of all datasets.","marker":"OpenAI, 2024"},{"why":"Supplies the research-titles dataset whose expert criteria serve as one of the two ground-truth benchmarks.","marker":"Wang et al., 2018"},{"why":"Supplies the advertising-slogans dataset used as the second ground-truth benchmark.","marker":"Jin et al., 2023"},{"why":"Defines the expert criteria for good research titles that DSAI must rediscover.","marker":"Nair and Gibbert, 2016"},{"why":"Defines the expert slogan guidelines that serve as ground-truth features for the slogans benchmark.","marker":"Kohli et al., 2007"}],"fun_headline_variants":["DSAI scores each feature's true discriminative power","LLM features grounded in data, not pre-trained knowledge","Unbiased latent features via prominence-scored criteria","Data-centric pipeline stops LLM feature regurgitation","Prominence metric reveals features that read the data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every prominence score, DP score, and grounding comparison in the paper is computed from positive and negative labels produced by GPT-4o after prompt-tuning against only 10 to 20 manually annotated samples; if those model-generated labels are noisy or biased, every downstream metric inherits that noise.","fun_headline_variants_meta":{"raw":{"variants":["DSAI scores each feature's true discriminative power","LLM features grounded in data, not pre-trained knowledge","Unbiased latent features via prominence-scored criteria","Data-centric pipeline stops LLM feature regurgitation","Prominence metric reveals features that read the data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1885,"prompt_tokens":990,"completion_tokens":895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":820}},"tokens_in":606,"tokens_out":895,"duration_ms":8252,"temperature":1.0,"reasoning_tokens":820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:25.690707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DSAI on a dataset whose true class structure contradicts GPT-4o's priors - for example, a slogan corpus where human experts rate clichéd, salesy slogans as 'good' and clever understated ones as 'bad' - and check whether the extracted features track the actual labels or drift toward the model's prior notion of good slogans; equivalently, feed DSAI the same labeled data with positive and negative labels swapped and see whether its features invert.","supporting_citations":[],"review_version":1}