{"id":"bc9a6162-ec59-4159-bd18-45d12dad908a","arxiv_id":"2412.05655","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new English-language NER dataset for car adverts (Condition, Historic, Sales Options) shows LLMs outperform fine-tuned encoders, with GPT-4o reaching 62 F1 vs BERT's 32.","lead":"This paper introduces Auto-AdvER, a new named entity recognition dataset and three-label schema for car advertisements, and benchmarks eight models on it. Decoder-only LLMs like GPT-4o score roughly twice the F1 of smaller encoder models like BERT on this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-vs-encoder ranking may be an artifact of the template-heavy trader sample; per-source evaluation is needed.","rationale":"The paper's contribution is a new domain NER dataset plus a model ranking on that dataset. Of the two claims highlighted by the reader, the 'first-ever English-language schema' claim is a literature claim that is hard to falsify from the paper alone, but it is not the load-bearing part of the empirical contribution. The empirical claim depends on the dataset representing car advertisements. Section 3.3 shows the trader half was deliberately enriched for template-style ads by selecting highest-volume dealers, which is not a random sample of the genre. The private half is regionally stratified, but the overall 605-ad mix is therefore not representative of the full distribution of car advertisements. The model comparison is then run only on this particular mix. ICL-based LLMs can exploit repetitive templates more easily than small fine-tuned encoders on a small training split, so the reported F1 gap may be an artifact of the sampling design. The paper reports no per-source breakdown, so there is no evidence the ranking holds outside the high-volume trader/template subcorpus. This is exactly the reader's weakest_assumption, and I agree with it. The 86/82 inter-annotator-agreement inconsistency and the absence of released data are also real concerns, but they are reproducibility and labeling issues rather than the central empirical claim. The proposed check, per-source evaluation after release, would settle whether the LLM-advantage conclusion generalizes beyond the current sample.","tokens_in":11007,"tokens_out":6795,"duration_ms":66267,"concrete_test":"After releasing the dataset with provenance labels, recompute Table 3 separately for trader-only and private-only test instances within the same three folds, or add a fourth fold of randomly sampled low-volume trader and private ads. Check whether GPT-4o's F1 (62.0) remains above DeBERTaV3-large (36.0) in both subsets by more than the reported standard deviations. If the gap shrinks below significance or reverses on private ads, the conclusion 'LLMs outperform encoder-only models' must be restricted to high-volume/template trader ads.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (Section 4.4) is that decoder-only LLMs outperform encoder-only models on the new Auto-AdvER dataset, with GPT-4o F1 62.0 vs BERT-base 32.0. The data used for that comparison, however, are not a representative sample of car advertisements: Section 3.3 states that trader ads were chosen by sorting traders by listing volume and selecting high-volume dealers 'as they typically follow a template', while private ads were stratified by UK region. This deliberately over-represents a narrow, repetitive template subgenre. LLM in-context learning (Section 4.3) tends to exploit such regular templates well, whereas the fine-tuned encoders were trained on only ~70% of 605 ads. The reported 30-point F1 gap may therefore reflect the sampling design rather than a general property of car-advertisement NER. The paper does not report per-source (trader vs private) results, so the claim as stated is not yet supported for the broader genre of car advertisements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Auto-AdvER, a three-label NER schema (Condition, Historic, Sales Options) for English-language car advertisements, developed through an industry-academia collaboration. The authors annotate 605 advertisements (104,382 tokens), report an inter-annotator agreement of 92% F1 from a double-annotated subset, and compare eight models—BERT-base/large, DeBERTaV3-base/large, GPT-4o, Gemini 1.5 Flash, Llama 3.1 70B, and Qwen 2.5 72B—in a three-fold setup. Their headline result is that decoder-only LLMs outperform the smaller encoder-only models, with GPT-4o achieving the best F1 (62.0) and DeBERTaV3-base the best encoder F1 (36.3). The paper also discusses downstream applications such as market dynamics, predictive maintenance, and consumer protection.","tokens_in":11183,"tokens_out":6875,"duration_ms":62933,"significance":"If the dataset, schema, and evaluation are made available, Auto-AdvER would fill a genuine gap: there is currently no established English-language car-advertisement NER benchmark, and the industry partnership gives the schema practical face validity. The manuscript has notable strengths: detailed annotation guidelines in Appendices A and B, explicit hyperparameters and model versions in Appendix C, a partial-match F1 protocol with an alignment step for LLM outputs, and a candid Limitations section acknowledging closed-source LLM contamination. The three-fold comparison of eight models is a useful data point for practitioners. However, the dataset is not yet released, the evaluation metric definition contains an algebraic inconsistency, the IAA count is internally contradictory, and the reported model ranking may be confounded by the deliberately template-heavy trader sample. These issues prevent the current version from fully supporting the central claims.","major_comments":[{"comment":"The equalities \"Precision = (COR + 0.5 × PAR)/ACT = TP/(TP+FP)\" and \"Recall = (COR + 0.5 × PAR)/POS = TP/(TP+FN)\" cannot both hold given the definitions of POS and ACT in the same paragraph: INC appears in the denominators but not in the numerators, and PAR is counted only half in the numerator. The reported F1 values therefore depend on an unstated decision about how incorrect-label and partial matches are treated. Please state explicitly whether the implementation used (COR + 0.5×PAR)/(COR+INC+PAR+SPU) and (COR + 0.5×PAR)/(COR+INC+PAR+MIS) or a TP/FP formulation, and recompute Table 3 if the formula changes.","section":"§4.1, Eqs. (2)–(3)"},{"comment":"The text states that 86 advertisements were double-annotated, while the caption of Table 2 says 82 double-annotated documents. In addition, the 92% agreement is a single point estimate from one annotator pair, with no confidence interval or variance information. Since the reliability of the schema is a central claim, the exact document count must be corrected and the IAA should be reported with a more informative analysis, such as bootstrap intervals or per-label counts.","section":"§4.2 and Table 2"},{"comment":"The sampling design deliberately over-represents template-based trader advertisements: the trader subsample was selected by sorting dealers by listing volume and choosing high-volume dealers \"as they typically follow a template.\" The paper itself notes a clear trader/private divide in §3.5, but Table 3 reports only aggregate results. The 30-point F1 gap between GPT-4o and BERT-base may be inflated by the template-heavy trader subset, since in-context learning tends to exploit such regular templates well. Please report trader versus private performance and discuss how the sampling design affects the conclusion that \"the class of LLMs outperforms the smaller encoder-only models.\"","section":"§3.3 and §4.4"},{"comment":"The label distribution is strongly imbalanced, with 2,134 Sales Options labels versus 573 Condition and 794 Historic labels. Aggregate F1 can be dominated by the majority label, so the reported model ranking may not hold for the minority labels. Please provide per-label precision, recall, and F1 for each model, or at minimum for the best encoder and the best LLM, so that readers can see whether the LLM advantage is uniform across Condition, Historic, and Sales Options.","section":"Table 3 and §3.5"},{"comment":"The dataset is described as \"will be made available upon publication,\" but no repository link, data sample, or complete annotation guideline document is provided in the manuscript. For a contribution whose central deliverable is a new dataset, this prevents readers from using or validating the resource. A permanent link with a stable version, or an explicit statement of proprietary restrictions, and the complete annotation guidelines should be included in the revised manuscript.","section":"§3.5 and footnote 1"}],"minor_comments":[{"comment":"The word \"consesus\" should be \"consensus.\"","section":"§3.2"},{"comment":"The statement \"we employed 100 samples from the training data in an in-context learning (ICL) scenario\" is ambiguous: it is not clear whether this means 100 examples per label or 100 examples in total. This detail is important for reproducing the LLM evaluation and for judging the fairness of the comparison.","section":"§4.3"},{"comment":"The Historic label definition includes \"new tyres, brake discs\" under component change, while the Condition label also covers tyre condition (e.g., \"tyre condition\"). The boundary between these two labels should be clarified, since it directly affects the reported inter-annotator agreement.","section":"Appendix A"},{"comment":"The claim \"first-ever English-language annotation schema and dataset for car advertisements\" is strong. Since only one prior automotive NER study (Hu and Ma, 2024) is cited, please add a brief statement of the literature search used to support the claim, or soften the wording.","section":"§6"},{"comment":"The row percentages do not all sum to 100 due to rounding (e.g., CONDITION sums to 99). Reporting raw counts in addition to percentages would remove ambiguity.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central contribution is a dataset, but the manuscript currently withholds the data and contains an inconsistency in the IAA count. I recommend requiring the authors to release the data or state restrictions, to resolve the metric equations in §4.1, and to add per-source and per-label results. The sampling issue in §3.3 is the most consequential for the claimed LLM-versus-encoder ranking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful benchmark/resource paper, not a conceptual breakthrough. The Auto-AdvER dataset is the first English-language NER schema for car advertisements as far as I can tell, and the three-label schema (Condition, Historic, Sales Options) is sensible and grounded in industry needs. The annotation methodology is careful: action research, iterative guideline development with industry partners, and a documented label set with excluded categories. The IAA of 92% F1 is respectable, though it is computed on only about 86 ads.\n\nThe model comparison is a legitimate first pass. The results show a clear gap between decoder LLMs and fine-tuned encoders (GPT-4o 62.0 vs BERT-base 32.0), consistent with results in other noisy text domains. But the paper overreaches slightly when it implies this ranking holds for car advertisements in general. Section 3.3 says trader ads were deliberately chosen from high-volume dealers because they 'typically follow a template,' and the dataset is half trader, half private. The LLMs use 100 in-context examples while the encoders are fine-tuned on the training split, so this is not apples-to-apples; the template-heavy sample likely favors the LLMs' pattern-matching. The paper does not report per-source (trader vs private) results, so we cannot tell whether the ranking persists on the more diverse private ads. That is the biggest soft spot, and it is fixable.\n\nMinor issues: the text says 86 double-annotated ads but Table 2 says 82; there is no significance testing, only per-fold standard deviations; the dataset is promised but not yet released; and the IAA is a single point estimate. Dataset release matters because a benchmark paper without data is just a description.\n\nWho is this for? People working on domain-specific NER, especially noisy user-generated text, and anyone building analytics for automotive listings. It deserves a serious referee pass, but the authors should be asked to release the data, report per-source scores, and add significance testing. The 86/82 discrepancy needs a fix too.\n\nI would give it conditional acceptance: the core direction is sound, the resource is new, and the flaws are addressable.","headline":"A genuinely new domain NER dataset with a reasonable first model comparison, but the headline LLM-vs-encoder result is scoped to a template-heavy sample and needs per-source reporting before the claim stands for car ads broadly.","tokens_in":11731,"tokens_out":2830,"would_cite":false,"duration_ms":27114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 605-ad English dataset with a Condition–Historic–Sales Options schema is offered as the first NER benchmark for car advertisements, and GPT-4o tops it.","keywords":["named entity recognition","car advertisements","Auto-AdvER","annotation schema","large language models","encoder-only transformers","inter-annotator agreement","in-context learning"],"falsifier":"A direct test would be to sample a fresh corpus of car advertisements uniformly across all trader sizes and private sellers, run the same eight models under the same three-fold protocol, and compare the ranking; if a fine-tuned encoder-only model matches or beats GPT-4o, or if the LLM advantage shrinks, the paper's central empirical claim that decoder-only LLMs outperform encoder-only models for car-ad NER would be falsified.","tokens_in":10819,"feed_emoji":"🚗","tokens_out":8284,"duration_ms":68138,"temperature":0.7,"pith_summary":"This paper is trying to establish that car advertisements are a distinct text genre for named entity recognition (the task of finding and labelling key facts in text), with their own important facts — the vehicle's current condition, its past events, and the extras offered in the sale — and that a three-label schema built with industry input can capture them reliably. It introduces Auto-AdvER, a 605-ad English dataset annotated with Condition, Historic, and Sales Options labels, and reports 92% partial-match F1 between annotators. On that dataset, decoder-only large language models outperform encoder-only transformers, with GPT-4o reaching 62.0 F1 compared with 32.0 for BERT-base, though all models remain far from perfect. If these results hold, the automotive industry gains a reusable benchmark and a data foundation for downstream work such as entity linking, market analysis, and predictive maintenance.","feed_headline":"Car-ad texts get a first fact-extraction benchmark; GPT-4o tops it","feed_subtitle":"A 605-ad dataset labels condition, history, and sales offers; large language models beat smaller BERT-scale models.","key_machinery":"The carrying mechanism is the Auto-AdvER annotation schema, a three-label scheme — Condition (present state of the car), Historic (past events affecting it), and Sales Options (tangible offers or services beyond the car itself) — with detailed span definitions and exclusion rules produced through DevOps-style iterations, an iterative collaboration model in which academic and industry teams alternate design and feedback. Evaluation runs on partial-match F1 from Segura-Bedmar et al. (2013), which gives half credit for overlapping spans and is used for both inter-annotator agreement and model ranking. For the decoder-only LLMs, the paper uses an in-context learning protocol with 100 training examples, label definitions in the prompt, and @@/## delimiters to mark entities, while encoder-only models are fine-tuned for token classification.","core_discovery":"The paper claims to be the first to offer an English-language annotation schema and dataset for car advertisements, created through an industry-academia collaboration. Its central empirical finding is that on this dataset decoder-only LLMs beat encoder-only models: GPT-4o scores 62.0 F1, Gemini 1.5 Flash 58.7, Llama 3.1 70B 55.7, and Qwen2.5 72B 47.7, while fine-tuned BERT and DeBERTaV3 models range from 32.0 to 36.3 F1. The authors present this not as a solved task but as a first benchmark: the schema is intended to be extended, and the labels to be refined into finer entities through entity linking. The paper also notes that closed-source LLM results should be read with caution because undisclosed training data may contaminate evaluations.","pith_inferences":["A natural extension the authors only hint at: because Qwen2.5 72B has the highest precision (75.7) while GPT-4o has the highest recall (54.7), a voting or ensemble system combining them could plausibly beat either single model; the paper lists boosting and voting as possible improvements but does not test this specific combination.","The exclusion rules in Appendix B — no optional or standard features, no custom-fitted parts, no car identifiables — mean the benchmark deliberately sidesteps some price-relevant information; a testable extension is to add a Features label and see whether LLMs can handle it consistently.","The 92% inter-annotator agreement uses partial-match F1, which gives 0.5 credit for overlapping spans; a stricter boundary-exact F1 might be lower, so consumers of the benchmark should report both metrics.","Because traders were sampled by listing volume and private sellers by region, the schema's portability beyond the UK and beyond template-based dealership ads remains untested; re-applying the guidelines to US or EU car adverts would clarify how much of the benchmark is genre-specific versus region-specific."],"forward_implications":["The three-label schema gives the automotive industry a common vocabulary for extracting facts from car ads, and the dataset can serve as a benchmark for future NER work in the genre.","For organizations choosing a model, the results suggest decoder-only LLMs, particularly GPT-4o-class systems, are a stronger starting point than fine-tuned BERT-scale encoders when labelled data are limited.","The dataset's noisy, telegraphic language positions it alongside WNUT16 and Twitter NER, making it a candidate stress test for robustness to non-standard user-generated text.","The authors' planned entity linking would turn coarse labels into fine-grained slots (e.g., number of owners, service history, warranty length), enabling market-dynamics and predictive-maintenance applications described in the paper."],"supporting_citations":[{"why":"The only earlier automotive-domain NER study, on Chinese car-accessory recognition; it defines the gap this paper fills.","marker":"Hu and Ma (2024)"},{"why":"Supplies BERT base and large, the encoder-only baseline that scores lowest in the comparison.","marker":"Devlin et al. (2019)"},{"why":"Supplies DeBERTaV3 base and large, the strongest encoder-only models in the comparison.","marker":"He et al. (2021a)"},{"why":"Source of GPT-4o, the closed decoder LLM with the highest F1.","marker":"OpenAI (2024)"},{"why":"Source of Gemini 1.5 Flash, one of the closed decoder LLMs evaluated.","marker":"Gemini Team et al. (2024a)"},{"why":"Source of Llama 3.1 70B, one of the open decoder LLMs evaluated.","marker":"Llama Team et al. (2024b)"},{"why":"Source of Qwen2.5 72B, the open decoder LLM with the highest precision.","marker":"Yang et al. (2024)"},{"why":"Provides the in-context learning NER prompting protocol used for all decoder LLMs.","marker":"Wang et al. (2023)"},{"why":"Defines the partial-match F1 metric used for inter-annotator agreement and model evaluation.","marker":"Segura-Bedmar et al. (2013)"},{"why":"WNUT16 is the closest comparable noisy-text NER benchmark used to characterise the data.","marker":"Strauss et al. (2016)"}],"fun_headline_variants":["First car-ad NER benchmark: LLMs outshine BERT-scale models","Car-ad NER: GPT-4o leads, but LLMs still imperfect","Auto-AdvER: first car-ad dataset, LLMs beat encoders","New NER for car ads: LLMs outperform fine-tuned BERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's usefulness depends on its 605-ad sample representing car advertisements broadly, but traders were chosen by listing volume, favouring template texts, and private ads were stratified only by UK region, so a different sample composition could change both the model ranking and the schema's value.","fun_headline_variants_meta":{"raw":{"variants":["First car-ad NER benchmark: LLMs outshine BERT-scale models","Car-ad NER: GPT-4o leads, but LLMs still imperfect","Auto-AdvER: first car-ad dataset, LLMs beat encoders","New NER for car ads: LLMs outperform fine-tuned BERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1828,"prompt_tokens":980,"completion_tokens":848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":764}},"tokens_in":596,"tokens_out":848,"duration_ms":12039,"temperature":1.0,"reasoning_tokens":764,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:29:13.593695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to sample a fresh corpus of car advertisements uniformly across all trader sizes and private sellers, run the same eight models under the same three-fold protocol, and compare the ranking; if a fine-tuned encoder-only model matches or beats GPT-4o, or if the LLM advantage shrinks, the paper's central empirical claim that decoder-only LLMs outperform encoder-only models for car-ad NER would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"WNUT16 is the closest comparable noisy-text NER benchmark used to characterise the data."}],"review_version":1}