{"id":"c878654b-9f33-4415-aa89-b323f8f7c470","arxiv_id":"2505.02851","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM-based pipeline filtered 14,746 web pages into 3,531 deduplicated 30-day challenge ideas and showed app-user satisfaction of 4.3 out of 5.","lead":"A pipeline called 30DayGen uses LLMs to filter web pages, extract structured 30-day habit challenges, and remove duplicates, producing 3,531 challenges for a habit-formation app. The result is a practical demonstration that LLM-driven curation can build a usable niche-domain corpus, with caveats about how the quality numbers were measured.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Search evaluation is circular: ground truth for each query is the system's own top-50 embedding results, so relevant challenges outside that candidate set are invisible and hit@3/NDCG are conditional on the retriever being complete.","rationale":"The central claim of the paper is that an LLM-driven pipeline can build a high-quality, searchable corpus for a niche domain, with hit@3=85% as headline evidence. For that claim to hold, the search evaluation must measure whether the system surfaces relevant challenges from the entire corpus, not just from a candidate set the system itself produced. Section 6.2.1's ground truth is generated by the same embedding retrieval used at runtime, so the evaluation is blind to any relevant challenge that the embedding model fails to place in the top-50. This is not merely a missing baseline; it directly caps what the metrics can see. The user study (Section 6.3) provides independent evidence of usability, but it is a subjective survey with 119 self-selected respondents and does not quantify retrieval completeness, so it does not repair the circularity. The deduplication evaluation has a related issue (Section 6.1.1 estimates recall only over top-5 embedding neighbors), but the search evaluation is more load-bearing because hit@3/NDCG are the paper's most prominent quantitative results and the basis for the conclusion that the corpus is searchable 'with high quality.' The reader's weakest assumption identified exactly this point; no new concern emerged from this pass. The appropriate response is to require the authors to re-run the search evaluation with an external, retrieval-independent ground truth before accepting the headline numbers, which is what a CONDITIONAL verdict already asks for. Hence no verdict change.","tokens_in":12000,"tokens_out":6493,"duration_ms":61413,"concrete_test":"Sample 30 queries (10 per specificity) from the evaluation set. For each, build an expanded candidate pool: the original top-50 text-embedding-3-large results, top-50 results from an independent embedding model (e.g., BGE or text-embedding-3-small), top-50 BM25 matches over titles and daily actions, and 200 challenges randomly sampled from the full corpus. Two annotators independently label each pooled challenge as relevant/helpful or not. Recompute hit@3, Precision@20, and NDCG using the expanded pool as ground truth. If newly relevant challenges are found outside the original top-50 and hit@3 drops materially (e.g., by more than 0.05), the reported 0.848 is inflated by the self-generated candidate set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2.1 constructs the ground truth for each search query by querying ChallengeDB with the query embedding and manually reviewing the top-50 most similar challenges. The runtime retriever (Section 5.1) uses the same embedding model (OpenAI text-embedding-3-large) to form its candidate set, and the Indexing component reuses those embeddings (Section 4.4). Any helpful challenge that is not among the top-50 embedding neighbors of the query is therefore absent from the labeled candidate set and can never be scored as relevant, so it cannot count against the system no matter how the reranker and validation behave. Hit@3, Precision@K, Recall@K, and NDCG are all computed relative to this self-generated pool; they measure reranking and validation quality conditional on the embedding retriever's candidate set being complete, not end-to-end search over the full 3,531-challenge corpus. The paper notes that 'majority of questions have fewer than 50 answers' (Section 6.2.1), but only among the top-50 embedding results; it never checks for relevant challenges outside that set. Since the headline 'hit@3=85%' (Section 7) is the primary quantitative evidence for the central claim, this circularity is the most load-bearing weakness: if the embedding model misses relevant challenges for many queries, the reported numbers overstate how well users can discover relevant content in the corpus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents 30DayGen, an end-to-end LLM-driven pipeline for constructing a niche-domain content corpus of 30-day habit-formation challenges. The offline component collects webpages through SERP queries, filters them with an LLM prompt, extracts structured 'wish' and 'daily action' fields with Gemini 2.0 Flash, deduplicates challenges via embedding similarity, LLM pair matching, and greedy correlation clustering, and indexes the result in ChallengeDB. The runtime component encodes a user wish, retrieves by cosine similarity, reranks with bge-reranker-v2-m3, and applies LLM-based validation. The paper reports 14,746 webpages processed, 3,531 unique challenges, deduplication F1 of 0.890, search hit@3 of 0.848 and NDCG of 0.797, and a user study (n=119) with mean satisfaction 4.3/5 and 91% willingness to use the search results. The authors claim this is the first end-to-end LLM-based data curation pipeline for user-facing niche content.","tokens_in":12232,"tokens_out":3415,"duration_ms":36342,"significance":"If the reported results hold, the paper makes a useful practical contribution: it demonstrates that LLM prompting, embeddings, and clustering can largely automate a curation task that traditionally requires substantial manual effort, and it ships real external evidence in the form of a deployed app, a public repository of prompts and data, and a user survey. The pipeline is described in sufficient detail to be reproduced, and the ablation of the LLM validation component is a sound way to isolate that component's contribution. The main quantitative claims are, however, conditional on evaluation-design choices that need scrutiny before the headline numbers can be taken at face value.","major_comments":[{"comment":"The search evaluation is circular. The ground truth for each query is constructed by embedding the query, retrieving the top-50 most similar challenges from ChallengeDB, and manually labeling those candidates, while the runtime retriever described in §5.1 uses the same embedding model (text-embedding-3-large) and the same index to form its candidate set. Any relevant challenge outside the top-50 embedding neighbors is therefore invisible to the ground truth and can never be scored as correct, so Hit@3, Precision@K, Recall@K, and NDCG measure reranking and validation quality conditional on the embedding retriever's candidate pool, not end-to-end search over the full 3,531-challenge corpus. The statement that 'majority of questions have fewer than 50 answers' (§6.2.1) only indicates that the judged pool is small; it does not establish that the pool is complete. The comparison against the no-filtering ablation remains meaningful under the same candidate pool, but the headline claim 'hit@3=85%' (§7) cannot be read as an end-to-end retrieval evaluation without an independent pooling strategy, such as pooling candidates from multiple heterogeneous retrievers, sampling challenges from outside the top-50, or obtaining relevance judgments on a random corpus sample.","section":"§6.2.1, Table 2, §7"},{"comment":"The deduplication thresholds (0.625 and 0.7) are selected by manually examining pairs in similarity ranges on the same dataset that is later used to compute the reported precision, recall, and F1. This is threshold tuning on the test data, so the F1=0.890 in Table 1 is likely optimistic. Additionally, the recall estimate itself relies on the same embedding similarity that powers the deduplication: for each of the 100 remaining challenges, only the top-5 most similar challenges are examined, so any duplicate outside that top-5 set is invisible to the manual annotation. This makes the recall estimate an upper-bound-style quantity rather than an unbiased estimate. The authors should report threshold selection on a held-out validation set or via cross-validation, and should estimate recall using a more complete annotation procedure, for example by pooling candidate duplicates from multiple similarity measures or by manually scanning a stratified sample of the original challenge list.","section":"§4.3 Step 3, §6.1.1, Table 1"},{"comment":"The search query set consists of 100 queries generated by Gemini 2.5 Pro Exp and labeled by the authors, with no inter-annotator agreement reported and no evidence that the query distribution matches real user behavior. The user study provides complementary evidence, but the self-reported Likert scores and willingness-to-use responses do not directly validate the offline NDCG or hit@3 numbers. The authors should report annotation guidelines, provide agreement statistics on a subset of queries, and ideally compare the offline query distribution with logged user wishes from the deployed app. This would clarify how representative the benchmark is of actual usage.","section":"§6.2.1, §6.3"}],"minor_comments":[{"comment":"In the text 'Table 5a' appears to refer to Figure 5(a); the table is not numbered as a table. Please correct the cross-reference.","section":"§6.3, Figure 5a"},{"comment":"The abstract reports 91% willingness and a 4.3 satisfaction score, while §1.3 reports 89% satisfaction and 91% willingness. Figure 2(a) and 2(b) show 91% and 89% respectively. Please reconcile these numbers and state explicitly which figure corresponds to which statistic.","section":"Abstract, §1.3, Figure 2"},{"comment":"The phrase 'curating content corpus to torso to tail domains' appears to be a typo; presumably 'long-tail domains' or 'head-to-tail domains' was intended.","section":"§7"},{"comment":"The definition of Recall@K ('where the number is computed as min(k, num_of_correct_results)') is ambiguous. Please specify whether recall is normalized by the number of relevant challenges in the ground truth, by K, or by min(K, |relevant|), since the current wording could be read in multiple ways.","section":"§6.2.1, Metrics"},{"comment":"The URL filter precision is measured on 100 removed webpages, but it would be useful to also report the false-negative rate on the 953 kept pages, since a filter can achieve high precision while discarding a large fraction of valid pages.","section":"§4.1 Step 3, §6.1.3"}],"recommendation":"major_revision","confidential_remarks":"The core contribution is plausible and the system is real, but the two most important quantitative claims—deduplication F1 and search hit@3—are both affected by evaluation-design issues that are fixable within the scope of a revision. I would encourage the editor to treat the search-evaluation circularity as the primary condition for acceptance; the authors should either re-run the evaluation with an independent pooling strategy or explicitly rescope the claims to 'reranking and validation quality given embedding retrieval'. The deduplication thresholds should also be justified on held-out data. The paper is otherwise a solid workshop-level contribution with useful practical details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: this paper is worth reading as an applied data curation contribution, and it deserves a real referee, but the quantitative headline (hit@3=85%, dedup F1=0.890) should be treated as upper bounds rather than settled measurements. The paper describes a real system, 30DayGen, that builds a 3,531-challenge habit corpus from 14k webpages, and it ships a deployed app with a 119-person user survey scoring 4.3/5. That user evidence is independent of the offline metrics and is the paper's strongest asset.\n\nWhat's genuinely new is the integration: web filtering, LLM extraction, a dedup recipe that combines embedding similarity with LLM judgment and greedy correlation clustering, and an LLM validation step at search time. Prior LLM corpus pipelines target QA or training data; this one targets user-facing curated content where uniqueness and structure matter. That framing is fair and the engineering is real.\n\nThe soft spots are real and mostly in the evaluation, not the core idea. First, the search ground truth is constructed by querying ChallengeDB with the same embedding model used at runtime and manually reviewing the top-50 results. Any relevant challenge outside that pool is invisible, so hit@3 and NDCG measure reranking and validation quality conditional on the retriever being complete—they do not measure end-to-end search over the full corpus. That's the load-bearing weakness, and the paper does not attempt to estimate recall beyond the top-50 pool. Second, dedup thresholds (0.625 and 0.7) appear to be picked on the same data used for the reported F1, and recall is estimated by checking only each retained challenge's top-5 embedding neighbors. Both choices inflate the reported numbers to some degree. Third, there are no error bars or inter-annotator agreement on the manual labels, and the appendix points to a GitHub repo but no committed artifact or hash, so the exact pipeline is not fully reproducible. None of these flaws sink the existence claim—the app works and users like it—but they mean the precise numbers are not trustworthy as published.\n\nWho should read it: applied data-curation practitioners and anyone building niche content corpora with LLMs. It's a good example of what \"good enough\" looks like in a real product setting. For peer review, I'd send it out and ask for three fixes: hold out threshold selection, build a ground truth that at least samples outside the retriever's top-50, and release the repo with a commit hash. With those, I'd be comfortable with it.\n\nCandidly, I'd rather see this paper revised and published than desk-rejected.","headline":"A real, deployed pipeline with a convincing user survey, but the headline numbers on dedup and search are conditional on a circular evaluation and should be treated as upper bounds.","tokens_in":12840,"tokens_out":2564,"would_cite":true,"duration_ms":24914,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that an LLM-driven pipeline can build a curated corpus of 3,531 unique 30-day challenges from 14,746 webpages in under two weeks.","keywords":["Large Language Models (LLMs)","Content Corpus","Habit Formation","Data Curation","Structured Content Extraction","Semantic Deduplication","Entity Linkage","User Study"],"falsifier":"Take a sample of the 100 test queries and have annotators independently read the full 3,531-challenge corpus, marking every challenge they find relevant and helpful without seeing the embedding-ranked candidate list; recompute hit@3, recall and NDCG against this exhaustive gold standard. If a substantial share of accepted challenges lies outside the embedding-top-50 sets, the reported 85% hit@3 and NDCG 0.80 will decline.","tokens_in":11716,"feed_emoji":"📚","tokens_out":9946,"duration_ms":89365,"temperature":0.7,"pith_summary":"The paper claims that a single LLM-driven pipeline can replace most of the manual labor of building specialized, user-facing content corpora from unstructured web sources. It validates this with 30DayGen, an end-to-end system that collects web search results, uses LLM prompting with few-shot examples to filter useful pages, extracts structured 'wish plus daily action' challenges from blog text, and removes near-duplicate entries using embedding similarity combined with LLM judgment. On the habit-formation domain, the pipeline reduced 14,746 candidate webpages to 953 useful ones, extracted 11,792 challenges, deduplicated them to 3,531 unique challenges, and achieved a hit@3 of 85% and an NDCG of 0.80 in offline search evaluation, along with a 4.3/5 user satisfaction score. If the central claim is right, the expensive and error-prone curation step for niche-domain applications can be largely automated and completed in under two weeks.","feed_headline":"LLMs turn 14,746 webpages into 3,531 curated challenges","feed_subtitle":"An end-to-end pipeline curates a user-facing challenge corpus in under two weeks, with 85% hit@3 at search time.","key_machinery":"The load-bearing mechanism is the 30DayGen pipeline itself, especially its three LLM-enhanced curation stages: few-shot URL filtering that scores pages 0-10 for usefulness; schema-guided extraction that turns unstructured blog text into 'wish' and 'daily action' fields; and a progressive deduplication stage that filters high-string-similarity duplicates, scores remaining pairs by embedding similarity (using a fast vector index), sends only the ambiguous middle band of pairs to an LLM for semantic judgment, and closes clusters with a greedy approximation of correlation clustering—a grouping method that places an item into the cluster it matches most, provided it matches at least half the cluster's members. The same embeddings power the runtime search, which retrieves candidates by cosine similarity, reranks them, and then asks an LLM to validate whether each candidate actually helps the user's stated wish. This machinery is what carries the claim because each stage removes a specific kind of noise—irrelevant pages, unstructured formatting, near-duplicate action plans, and semantically adjacent but unhelpful suggestions.","core_discovery":"On its own terms, the paper establishes that an LLM-centered pipeline can take a niche domain—habit formation—from raw web search results to a clean, structured, searchable corpus of user-facing content without manual curation at each step. The key empirical results are: LLM URL filtering reaches 94% precision at selecting pages that actually contain 30-day challenge ideas; LLM extraction produces 11,792 structured challenges from 953 pages; a progressive deduplication method (string pre-filter, embedding similarity, LLM matching for ambiguous pairs, then greedy correlation clustering) removes duplicates with an F-measure of 0.890; and runtime search that couples embedding retrieval with LLM validation reaches hit@3 = 85%, with the validation step adding precision on highly specific queries. The authors present this as the first end-to-end LLM-based data curation pipeline for user-facing niche content.","pith_inferences":["My inference: the strong numbers depend on a ground truth built with the same embedding model used for retrieval, so an independent human-built gold standard could revise hit@3 and NDCG downward; that comparison is the natural next experiment.","My inference: the dedup design is a three-stage cost filter that spends LLM judgment only on pairs where embedding similarity is ambiguous, so the method should scale to corpora much larger than 11,792 entries without linear LLM cost growth.","My inference: the transferable contribution is the staged filter-plus-judgment architecture, not the particular similarity thresholds; any new domain will need recalibration of the middle threshold band and the cluster-membership rule.","My inference: if the pipeline is applied to a second domain with fresh human labels, achieving similar F1 and hit@3 would indicate the mechanism generalizes; failing to would point to domain-specific tuning being essential."],"forward_implications":["Building a niche content corpus from the web can drop from weeks of manual editing to an automated pipeline running in under two weeks.","The same architecture transfers to other long-tail domains with small instance counts, such as podcasts, online courses, self-help resources, and recipe collections, as the authors suggest.","Semantic deduplication of action plans—recognizing that 'cook a new meal daily' and 'try a new recipe daily' are the same challenge—is feasible with embedding similarity plus LLM judgment, not just string matching.","LLM validation at search time filters suggestions that are thematically close but actively counterproductive, improving precision on ultra-specific user wishes."],"supporting_citations":[{"why":"Grounds the premise that the web can serve as a corpus for building specialized text collections.","marker":"[10]"},{"why":"Supplies the blocking–matching–clustering paradigm for entity resolution that the deduplication stage adapts.","marker":"[5]"},{"why":"Supplies the fast vector similarity index used to generate duplicate-pair candidates from embeddings.","marker":"[13]"},{"why":"Defines correlation clustering, the NP-hard grouping objective that the greedy clustering step approximates.","marker":"[1]"},{"why":"Shows that LLMs can improve entity matching over string-based methods, motivating the LLM matching stage for ambiguous duplicate pairs.","marker":"[15]"},{"why":"Defines NDCG, the ranking metric used in the search evaluation.","marker":"[12]"}],"fun_headline_variants":["LLM pipeline turns 15K webpages into 3,531 curated challenges","AI curation: 15K webpages become 3,531 structured habit challenges","LLM pipeline: 3,531 habit challenges from 15K webpages","Niche corpus creation: LLMs curate 3,531 challenges from web"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the ground truth for each search query—the 50 most embedding-similar challenges, manually reviewed—contains every challenge a user would consider relevant, so any relevant challenge the embedding step fails to surface is invisible to the reported precision, recall, and NDCG.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline turns 15K webpages into 3,531 curated challenges","AI curation: 15K webpages become 3,531 structured habit challenges","LLM pipeline: 3,531 habit challenges from 15K webpages","Niche corpus creation: LLMs curate 3,531 challenges from web"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2519,"prompt_tokens":893,"completion_tokens":1626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":509,"tokens_out":1626,"duration_ms":11293,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:25:58.199133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of the 100 test queries and have annotators independently read the full 3,531-challenge corpus, marking every challenge they find relevant and helpful without seeing the embedding-ranked candidate list; recompute hit@3, recall and NDCG against this exhaustive gold standard. If a substantial share of accepted challenges lies outside the embedding-top-50 sets, the reported 85% hit@3 and NDCG 0.80 will decline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the premise that the web can serve as a corpus for building specialized text collections."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the blocking–matching–clustering paradigm for entity resolution that the deduplication stage adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines correlation clustering, the NP-hard grouping objective that the greedy clustering step approximates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines NDCG, the ranking metric used in the search evaluation."}],"review_version":1}