{"id":"e4ac4f33-23ff-4542-99b4-c1a4a128a67d","arxiv_id":"1908.04364","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A dataset of 923k Amazon questions, 14M reviews, and 3.6M human-written answers for review-based question answering, with answerability labels for each question.","lead":"AmazonQA is a new dataset that gives computers a realistic task: answer a shopper's question about a product using that product's written reviews. The authors also label which questions can truly be answered from reviews and show that current question-answering models still struggle at this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's closed-world guarantee that answerable questions are answerable from the provided snippets is unenforced and contradicted by the paper's own IR-span evaluation, threatening benchmark validity.","rationale":"The reader's weakest assumption focuses on the answerability classifier: a logistic regression trained on 3,297 crowd instances and applied to 923k questions, with only a 137-instance expert test set. That is a genuine concern, and the paper even contains a numeric inconsistency (72% precision in §3.2 vs. 0.67 in Table 3). However, I see an even more fundamental load-bearing premise: the sufficiency of the provided review snippets as the closed-world context. The paper explicitly claims in Section 2 that for any answerable question the snippets contain all required supporting information. This guarantee is never verified; the snippets are selected by BM25, and the answerability classifier itself operates on the top-5 snippets without checking whether the actual answer is present. Table 4 provides internal evidence that the IR retrieval often fails: only 58% of questions are judged answerable by experts using IR-selected sentences. If this retrieval failure propagates to the full 10-snippet context, then a large portion of the 570k 'answerable' pairs may not actually be answerable from the distributed context, making the benchmark ill-posed for reading-comprehension-style evaluation. This concern subsumes the label-quality issue: noisy or biased answerability labels are a downstream consequence of insufficient and unverified context. The resource may still be valuable as a large collection of real review-QA interactions, but the paper's core claim—that it is a usable review-based QA benchmark with reliable support—requires positive evidence that the snippets actually contain the answers. A focused human evaluation on a fresh sample from the released data, as described in the concrete test, would settle this. I therefore keep the reader's CONDITIONAL verdict unchanged: the concern does not invalidate the dataset, but it demands a concrete validation step before the benchmark's closed-world promise can be accepted.","tokens_in":8956,"tokens_out":9407,"duration_ms":93321,"concrete_test":"Draw a random sample of 300 question-context pairs from the released 570k answerable training split. Have three expert annotators, given the question and the 10 provided snippets, independently label (a) whether the snippets contain enough information to answer the question, and (b) which snippet(s) support the answer. Compute the majority-agreement rate that the pair is answerable from the snippets, and also measure whether the Amazon reference answer is entailed by the snippets. If the answerable-from-snippets rate falls below 0.8 (or the reference-answer entailment rate is low), the closed-world guarantee in §2 is empirically false and the benchmark's context must be re-extracted or the claim retracted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states: 'we pose the final question-answering in a closed-world setting. We ensure that for any answerable question, the associated snippets contain all the required supporting information.' No construction step actually enforces this. The answerability label is a logistic-regression prediction on 8 features over the top-5 BM25 snippets (§4.2), not a verified entailment check. The paper's own Table 4 on a 100-question expert sample reports that only 0.58 of questions are answerable from IR-selected sentences, showing that BM25 retrieval frequently fails to surface the supporting evidence. Although Table 4 evaluates single sentences rather than the full 10 snippets, it directly measures the IR component that produces the snippets and the classifier context. If a large fraction of the 570k 'answerable' pairs lack the answer in the provided snippets, then (i) the closed-world setup is violated, so a model cannot be expected to produce the reference answer even with perfect reading comprehension; (ii) the answerability classifier is trained and evaluated on snippet-restricted evidence, so its labels inherit the same retrieval gaps; and (iii) the BLEU/ROUGE baselines in Table 7 penalize models for missing information that is not in the context. This is not just a matter of label noise; it threatens the core utility of the dataset as a reading-comprehension benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AmazonQA, a large-scale question-answering dataset built from Amazon product reviews, questions, and community answers. The authors propose the task of review-based QA: given a product's reviews and a question, generate a natural-language answer. The dataset comprises 923k questions, 3.6M answers, and 14M reviews across 156k products. To support supervised training, the authors extract top-10 BM25 review snippets per question, build an answerability classifier trained on 3,297 crowd-annotated instances (with a 137-instance expert test set), and filter to roughly 570k answerable question-answer-review instances. They split the data by product into train/development/test sets, and provide baseline experiments: three LSTM language models, heuristic sentence/review retrievers, and an R-Net span-based reading-comprehension model. The central claims are that AmazonQA is the largest public QA dataset with descriptive answers and the first large-scale benchmark where questions, support, and answers all come from real Amazon interactions.","tokens_in":9154,"tokens_out":2935,"duration_ms":32485,"significance":"If the dataset is sound, it is a valuable resource for the QA community. Its strengths are the realism of the data (questions, answers, and reviews are all organic user-generated content), the large scale, the product-based split that prevents answer memorization from question-product leakage, and the public release of data and baseline implementations. The paper also explicitly includes ablation-style language models (P(a), P(a|q), P(a|q,R)) to verify that reviews add information beyond the question alone. However, the paper's utility as a reading-comprehension benchmark rests on two load-bearing assumptions: that the answerability labels are reliable at scale, and that the provided review snippets actually contain the supporting evidence for answerable questions. The current manuscript provides only limited evidence for both, and some of the evidence it does provide (Table 4) points in the opposite direction. These issues are correctable with additional validation and revised claims, but they are central to the benchmark's validity rather than mere presentation concerns.","major_comments":[{"comment":"The abstract and §3.2 state that the answerability classifier 'marks roughly 570K pairs as answerable out of total of 923K question with 72% precision,' but Table 3 reports the classifier's precision as 0.67 on the expert test set. The 72% figure is not defined as a different operating point or a different evaluation subset. Because the abstract advertises precision as a headline property of the released dataset, this inconsistency must be resolved by reporting the exact threshold and evaluation set that produce each number.","section":"Abstract and §3.2"},{"comment":"Section 2 claims: 'We ensure that for any answerable question, the associated snippets contain all the required supporting information.' No construction step enforces this guarantee. The answerability label is produced by a logistic-regression classifier over eight features computed from the top-5 BM25 snippets (§4.2), not by a verified entailment check against the full review text or the provided 10 snippets. Table 4 directly measures the retrieval component and reports that only 0.58 of 100 expert-annotated questions are answerable from IR-selected sentences. Since the classifier's context is the same retrieval pipeline, a substantial fraction of the 570k 'answerable' instances likely lack the supporting evidence in the provided snippets. This invalidates the closed-world premise and means the BLEU/ROUGE baselines in Table 7 penalize models for information that is not in the context. The authors should either enforce the closed-world property through a verifiable filtering step, or explicitly reframe the dataset as open-world with noisy evidence and quantify the resulting label noise.","section":"§2 and Table 4"},{"comment":"The answerability classifier is trained on 3,297 crowd labels and then applied to the full 923k corpus with no additional validation on the target distribution. The only held-out expert evaluation consists of 137 test instances, which is too small to estimate error rates across product categories and question types. Because the utility of the 570k training instances depends on label quality, the paper should report category-wise precision/recall, the sensitivity of the labels to the decision threshold, and an additional expert-annotated sample from the deployed classifier's output to measure how label noise propagates into the training set.","section":"§4.2"},{"comment":"The evaluation metrics are applied uniformly to yes/no and descriptive questions, although the authors acknowledge in the text that BLEU/ROUGE are 'not the best metrics' for yes/no questions. The observation that a random sentence achieves nearly the same BLEU/ROUGE as Amazon user answers (78.56 vs. 80.88 BLEU-1) suggests these metrics may not be informative for this dataset. The paper proposes a combined metric set only as future work; given that the benchmark's usefulness depends on meaningful evaluation, the authors should at least report accuracy or F1 for the yes/no subset, or provide a more detailed analysis of metric validity, before claiming that the baselines demonstrate the task's difficulty.","section":"§6, Table 7"}],"minor_comments":[{"comment":"The snippet construction description ('snippets of length 100, or to the end of a sentence boundary, whichever is greater') is ambiguous: it should specify whether the snippet length is measured in tokens or characters, and how the 'greater' comparison is made between a fixed length and a sentence boundary.","section":"§3.1"},{"comment":"The entry for RACE lists '# Qs' as '97l', which appears to be a typo for a number with a letter 'l' instead of a digit; this should be corrected.","section":"Table 1"},{"comment":"The caption should state explicitly that the evaluation is on a 100-question test set and that 'IR' refers to BM25 sentence retrieval, so that the reader can connect this table to the snippet-construction pipeline in §3.1.","section":"Table 4"},{"comment":"The R-Net citation [Group, 2017] is a blog post rather than a peer-reviewed publication; the authors should cite the original technical report or a published version if one exists.","section":"§5.2"},{"comment":"The last column mixes two different quantities — P(a|q) and P(a|q,R) perplexities — under a single header, which is confusing; the column should be split or the header should clearly indicate that the two numbers correspond to different models.","section":"Table 7"},{"comment":"The abstract says 'we propose a method' but the paper primarily introduces a dataset and baseline models; the phrasing overstates the methodological novelty and should be aligned with the actual contribution.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset paper with genuine value, and the core resource is likely useful to the community even if some claims need to be softened. I recommend major revision rather than rejection because the main problems — the unmatched precision claim, the unenforced closed-world guarantee, and the small-scale validation of the answerability classifier — can all be addressed with additional analysis and revised wording. The authors should also be encouraged to release the exact preprocessing code and classifier threshold to make the dataset reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading: it contributes a large, real, publicly released dataset of 923k questions, 3.6M answers, and 14M reviews from Amazon, with answerability annotations and a product-level split that avoids leakage. That split is a thoughtful design choice, and the baseline study is a fair starting point, including an explicit caveat that BLEU/ROUGE are poor metrics for yes/no questions. The authors also provide code and baselines, which makes the resource immediately usable.\n\nThe soft spots are mostly around the answerability layer. The classifier is trained on 3,297 crowd-labeled instances, tested on 137 expert instances, then applied to the full 923k corpus. That is a thin basis for labeling two-thirds of the dataset. Worse, the abstract claims 72% precision while Table 3 reports 0.67; that inconsistency needs reconciling. The closed-world guarantee is also stated more strongly than the construction supports. The paper says the snippets contain all required supporting information, but snippets are BM25 picks and the answerability label is a logistic regression on 8 features, not an entailment check. The authors' own Table 4 shows only 0.58 of top-1 IR sentences are considered answerable by experts; that is single-sentence rather than ten-snippet evidence, so it does not directly kill the dataset, but it does undercut the word \"ensure.\" A careful reader will want a larger validation set for the distributed labels.\n\nThe novelty claim also needs calibration. The related work cites Yu et al. (2012), which literally builds a system that answers product questions from consumer reviews, yet the text says no prior work does this. That is a factual overstatement, not a fatal one, but it should be fixed.\n\nFor whom is this paper? Researchers building e-commerce QA systems, or anyone who wants a large, noisy, realistic QA resource with free-form answers and a genuine closed-world problem. The dataset itself is the contribution, and the baselines are useful for measuring progress.\n\nI would send this to peer review: the resource is real and the issues are fixable. The referee should focus on the answerability validation, the precision discrepancy, and the closed-world claim. It is a solid dataset paper with some sloppy packaging.","headline":"A genuinely useful real-world QA dataset, but the answerability layer and a couple of overclaims will need attention from a careful referee.","tokens_in":9744,"tokens_out":2857,"would_cite":true,"duration_ms":30124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes review-based question answering and introduces AmazonQA, a dataset of 923k questions, 3.6M answers, and 14M reviews across 156k products, arguing that many product questions can be answered by synthesizing information…","keywords":["question answering","product reviews","community question answering","dataset","reading comprehension","answerability classification","e-commerce","benchmark"],"falsifier":"Take a random sample of, say, 400 questions from the 570k 'answerable' pool, have independent experts label whether the provided review snippets truly answer each question, and compare against the classifier's labels; if expert precision is substantially below the reported 72% or errors correlate with question type or length, the dataset's central resource is not trustworthy.","tokens_in":8717,"feed_emoji":"📦","tokens_out":4523,"duration_ms":40117,"temperature":0.7,"pith_summary":"Customers ask thousands of questions on product pages, but answers often take days. The paper proposes that many of those questions could be answered immediately by reading the product's reviews, and formalizes this as review-based question answering: given a question and a review corpus, generate a natural-language answer. To support the task it introduces AmazonQA, with 923k questions, 3.6M answers, and 14M reviews across 156k products, and an answerability classifier that marks roughly 570k question-context pairs as answerable. The paper's claim is that this is the largest public QA dataset with descriptive (free-form) answers and the first large-scale benchmark where questions, supporting passages, and answers all come from real human interactions.","feed_headline":"923k real Amazon questions become a QA benchmark","feed_subtitle":"The largest public QA set with free-form answers, built from real reviews, questions, and answers, with an answerability filter.","key_machinery":"The load-bearing mechanism is a three-stage pipeline. First, reviews are chunked into snippets and ranked by BM25 to give each question its top ten candidate passages. Second, a logistic regression classifier, trained on crowd labels with tf-idf and GloVe-derived features, decides whether the reviews answer the question. Third, either an LSTM sequence-to-sequence model generates an answer conditioned on question and review snippets, or a span-based reader (R-Net) extracts an answer from heuristically generated spans. The answerability classifier is what converts raw crawled data into a supervised learning resource.","core_discovery":"The central discovery is a new benchmark and task formulation. The authors extract real customer questions, community answers, and product reviews from a large crawl, filter and clean them, and add answerability labels so that a QA system can first decide whether the reviews contain an answer and then generate one. They show that standard span-based reading comprehension models underperform simple retrieval heuristics like taking the top BM25-ranked sentence, and that conditioning answer generation on both the question and reviews lowers perplexity versus question-only or no-conditioning models. This establishes the task as challenging and not solved by existing methods.","pith_inferences":["Because yes/no questions make up 15% of the data, a system that answers them correctly might score poorly on BLEU/ROUGE; an accuracy metric on that subset would give a clearer signal than token-overlap metrics alone.","The answerability labels are the dataset's linchpin, and they come from a classifier applied at scale; users of AmazonQA should verify label quality on their own random sample before trusting the 570k answerable pairs.","The 17 product categories allow a natural out-of-domain experiment: train on some categories and test on held-out ones, which could reveal whether review-based QA systems learn transferable reading behavior.","Observing that even random review sentences score close to human answers suggests that many reference answers share vocabulary with the reviews; evaluation may need to control for answer redundancy to measure true understanding."],"forward_implications":["A deployed system could intercept a new product question, classify it as answerable from reviews, and return a provisional answer immediately instead of making the user wait days for a community response.","Because the questions, evidence, and answers are all real human interactions rather than artificially constructed, models trained on AmazonQA may reflect the actual difficulty of product QA better than synthetic benchmarks.","The product-based train/dev/test split forces models to answer unseen products from reviews alone, so performance measures genuine reading and synthesis rather than memorization of product-specific answers.","The result that top-sentence retrieval beats neural reading comprehension indicates that effective review-based QA may require better answer synthesis, not just better evidence selection."],"supporting_citations":[{"why":"Supplies the raw crawl of Amazon reviews, questions, and answers that AmazonQA is built on, plus the question-type annotation method.","marker":"[McAuley and Yang, 2016]"},{"why":"Provides the BM25 ranking used to extract the top review snippets for each question, forming the retrieval component of the pipeline.","marker":"[Robertson and Zaragoza, 2009]"},{"why":"Provides the GloVe embeddings used, weighted by tf-idf, as features in the answerability classifier.","marker":"[Pennington et al., 2014]"},{"why":"Provides R-Net, the span-based reading comprehension model trained on heuristically generated spans as the neural baseline.","marker":"[Group, 2017]"},{"why":"Defines the SQuAD span-based QA paradigm that motivates converting AmazonQA into a span format and comparing against reading comprehension models.","marker":"[Rajpurkar et al., 2016]"},{"why":"MS MARCO is the closest existing free-form answer dataset and serves as the comparison point for AmazonQA's novelty.","marker":"[Nguyen et al., 2016]"},{"why":"Motivates the ablation-style language model baselines that test whether conditioning on the question and reviews actually contributes to answer generation.","marker":"[Kaushik and Lipton, 2018]"}],"fun_headline_variants":["923k Amazon questions become review-based QA benchmark","Simple retrieval outperforms reader models on AmazonQA","Answerability filtering makes Amazon QA more realistic","Real human Q&A on 156k products becomes new QA challenge","Review-based QA: 923k questions, 3.6M answers, 14M reviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The full dataset's answerability labels are produced by a logistic regression classifier trained on only 3,297 crowd-annotated question-context pairs and checked on 137 expert labels; if that classifier is systematically wrong about which questions can be answered from reviews, the 570k pairs that form the core training resource are biased.","fun_headline_variants_meta":{"raw":{"variants":["923k Amazon questions become review-based QA benchmark","Simple retrieval outperforms reader models on AmazonQA","Answerability filtering makes Amazon QA more realistic","Real human Q&A on 156k products becomes new QA challenge","Review-based QA: 923k questions, 3.6M answers, 14M reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3304,"prompt_tokens":865,"completion_tokens":2439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2354}},"tokens_in":481,"tokens_out":2439,"duration_ms":18590,"temperature":1.0,"reasoning_tokens":2354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:43:57.259346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 400 questions from the 570k 'answerable' pool, have independent experts label whether the provided review snippets truly answer each question, and compare against the classifier's labels; if expert precision is substantially below the reported 72% or errors correlate with question type or length, the dataset's central resource is not trustworthy.","supporting_citations":[{"cited_title":"The probabilistic relevance framework: Bm25 and beyond","cited_arxiv_id":null,"evidence_quote":"Provides the BM25 ranking used to extract the top review snippets for each question, forming the retrieval component of the pipeline."},{"cited_title":"R-net: Machine reading comprehension with self-matching net- works","cited_arxiv_id":null,"evidence_quote":"Provides R-Net, the span-based reading comprehension model trained on heuristically generated spans as the neural baseline."},{"cited_title":"Ms marco: A human generated machine reading comprehension dataset","cited_arxiv_id":null,"evidence_quote":"MS MARCO is the closest existing free-form answer dataset and serves as the comparison point for AmazonQA's novelty."}],"review_version":1}