{"id":"54cb9890-7003-498e-a6d0-b28c7c550191","arxiv_id":"2507.12104","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AI4Pricing2Yaml is an LLM-based extractor that turns static HTML pricing pages into structured YAML pricing models, with reported mean accuracy of 88% for features but only 54% for add-ons.","lead":"This paper describes a tool that uses a large language model to read SaaS pricing pages and convert them into structured machine-readable pricing files. The authors tested it on 30 real pricing pages, reporting strong results for extracting features but weaker results for usage limits and add-ons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation credits unextracted, dynamically hidden data as True Negatives and omits 15 of 30 sites from element metrics; the abstract's 'high accuracy and recall' rests on that credit.","rationale":"The reader's weakest assumption correctly identifies the TN and gold-standard issue, and I agree with that diagnosis. The concern is load-bearing because the central claim is an aggregate accuracy/recall statement, and the two largest threats to it are both inside the metric construction rather than in the model's output quality. The paper deserves credit for releasing artifacts and for explicitly acknowledging dynamic-content and hallucination limits in Section 5; those acknowledgements do not repair the scoring definition, because the definition converts acknowledged failures into correct outcomes. The feature-extraction results (mean accuracy 88.2%, recall 96.4%) are genuinely encouraging, so this is not a rejection concern: if a conservative recomputation keeps recall high for features and plans, the extractor claim stands. No baselines are reported, but the absence of a baseline is secondary to the measurement problem; a baseline would not help if the metric itself credits missing data. The verdict remains CONDITIONAL: the paper should be accepted only after the authors recompute metrics conservatively and either qualify the 15-site limitation or show that the results are robust.","tokens_in":8981,"tokens_out":5007,"duration_ms":59546,"concrete_test":"Use the Zenodo raw outputs and the companion dataset from [7] to recompute Table 1 under two rules: (a) for the 15 SaaS whose features/usage limits/add-ons were not successfully extracted, place the gold-standard expected elements in the FN column instead of dropping those rows; (b) independently re-annotate the 30 live pricing pages with a headless browser that performs clicks/expansions, and treat any element that becomes visible after interaction as expected positive, so it cannot be counted as TN. If add-ons recall falls materially below the reported 81% (or usage-limit recall below 77.8%), or if accuracy drops by more than about 10 points, then the abstract's 'high accuracy and recall' claim is not supported. The same recomputation also reveals how many TN credits actually correspond to missing dynamic content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing weakness is the evaluation design in Section 4.1. True Negatives are defined as 'data not extracted because it was absent from the scraped information, although it might be accessible through dynamic interactions (e.g., by clicking a button...)'. The scoring system gives a full point for a TN and computes accuracy as (TP+TN)/total. This credits the extractor for exactly the content it failed to retrieve: dynamic content that would require clicks or modals is treated as correct non-extraction rather than as a false negative. The paper then states that only 15 of the 30 SaaS had features, usage limits, and add-ons successfully extracted; the other 15 are not counted as failures. Every headline metric in Fig. 3 inherits this inflation. The conclusion in Section 6 that the system 'efficiently extracts key components... achieving high accuracy and recall' therefore depends on a rubric that subtracts the hardest cases from the denominator and rewards the system for missing them. The gold standard is the authors' own manually built dataset and schema, which makes it difficult to detect whether elements behind dynamic interactions were ever included in the expected-positive set. A secondary overreach is that Section 6 claims automation of the full transformation while Section 3.2 states only the Information Extractor was implemented; the Process Engine and Results Modeler are not evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the notion of 'intelligent pricing' (iPricing) as a machine-readable, dynamically maintainable representation of SaaS pricing, and presents AI4Pricing2Yaml, an LLM-based system intended to transform static HTML pricing pages into this representation. The proposed architecture has three components (Information Extractor, Process Engine, Results Modeler), but the implementation and evaluation cover only the Information Extractor, which uses Selenium for scraping and Gemini 1.5 Flash with prompt engineering to extract plans, features, usage limits, and add-ons. The evaluation is performed on a dataset of 30 commercial SaaS websites taken from the authors' prior work, with metrics of accuracy, precision, and recall computed per element type. The paper reports high recall for plans and features, lower performance for usage limits and add-ons, and discusses challenges with dynamic content and hallucinations. The central claim is that the system automates the transformation of static pricing into intelligent pricing with high accuracy and recall.","tokens_in":9245,"tokens_out":5832,"duration_ms":64356,"significance":"If the claims were fully supported, the paper would offer a useful reusable component for SaaS pricing analysis. The strengths are the concrete implementation, the use of a real-world dataset of 30 commercial websites, the absence of fitted parameters (the approach relies on prompt engineering of a pre-trained LLM), and the availability of supplementary artifacts including raw outputs and code. However, the evaluation design in Section 4.1 is the load-bearing weakness: the True Negative definition credits the system for not retrieving dynamic content, and 15 of 30 sites are excluded from the per-element metrics. As a result, the headline accuracy and recall figures do not substantiate the abstract's claim that the system extracts the desired elements 'across all steps.' The paper is a reasonable proof-of-concept for the extraction step, but the full transformation claim and the quantitative strength of the results require substantial revision.","major_comments":[{"comment":"The definition of True Negatives ('Data not extracted because it was absent from the scraped information, although it might be accessible through dynamic interactions') credits the extractor for failing to retrieve content that exists behind clicks or modals. Because accuracy is computed as (TP+TN)/total, the system is rewarded exactly for the cases it misses. The text further states that only 15 of the 30 SaaS platforms had features, usage limits, and add-ons successfully extracted; the remaining 15 are not counted in the per-element means. The headline metrics (e.g., 88.2% feature accuracy, 53.5% add-on accuracy, and the corresponding recall values in Fig. 3) are therefore computed on a favorable subset and do not support the abstract's claim that the system extracts 'the desired elements across all steps' for the 30-site dataset. Please recompute the results treating unextracted dynamic content as false negatives, including all 30 sites in every per-element metric, and reporting accuracy both with and without TN contributions.","section":"Section 4.1, Table 1, Fig. 3"},{"comment":"Section 3.2 explicitly states that only the Information Extractor was fully implemented ('we have fully implemented the first one, the Information Extractor'), while the Process Engine and Results Modeler are part of the architecture but were not implemented or evaluated. Nevertheless, the abstract and Section 6 conclude that AI4Pricing2Yaml automates the transformation of static HTML pricing into iPricing and 'efficiently extracts key components... achieving high accuracy and recall.' This overclaims: the validation covers only the extraction component, not the full pipeline. Please either implement and evaluate the remaining components or explicitly restrict the conclusions to the extraction step.","section":"Section 3.2 vs. Section 6 and Abstract"},{"comment":"The point-based scoring system with half points is not a standard evaluation protocol and makes the reported metrics difficult to interpret. Awarding half a point to both the TP and FP categories for the same item (e.g., a feature extracted but marked as available in too many plans) counts the same evidence in both the numerator and denominator of precision and inflates the total number of extractions. Please provide raw TP/FP/FN/TN counts per site and per element without fractional credits, or define a multi-label loss function that justifies the scoring. The reported means should be reproducible from raw counts.","section":"Section 4.1, scoring system"},{"comment":"The phrase 'encompassing over 150 intelligent pricings' is not defined or derivable from Table 1. Table 1 reports per-element counts for 30 SaaS sites; no aggregation or definition shows where 150 comes from. Please clarify what counts as an 'intelligent pricing' in this number, and how it relates to the 30 websites and the extracted plans, features, usage limits, and add-ons.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The text contains the typo 'through apric-ing'; this should read 'through a pricing' or similar.","section":"Section 1"},{"comment":"The context window is given as 'up to 106 tokens'; this should read '10^6 tokens'.","section":"Section 3.2"},{"comment":"The sentence 'A full point is awarded for each correct extraction (TP or TN) and for incorrect extractions (FP or FN)' appears to be a typo: incorrect extractions should receive zero points, not a full point.","section":"Section 4.1"},{"comment":"Table 1 is very hard to read because the column headers repeat 'TP FP FN TN T A (%) P (%) R (%)' for each of the four element types without visual separation; please reformat the table or split it into separate tables per element type.","section":"Table 1"},{"comment":"The proposal of an 'ideal SaaS pricing webpage' is based on the authors' six-year analysis, but the text does not present evidence or criteria for how the ideal structure was derived; this should be stated as a recommendation rather than a validated finding.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a proof-of-concept for an LLM-based pricing extractor, and the authors are transparent about the limitations. The central problem is that the quantitative evaluation is constructed in a way that rewards the system for missing dynamic content, and the full transformation pipeline is not implemented. These issues are fixable by re-analyzing the data with stricter metrics and by revising the claims to match the implemented component. The self-reference burden is modest but present: the dataset, the schema, and the gold-standard annotations all come from the authors' own prior work, with no independent annotation. This is worth asking the authors to acknowledge more explicitly. The paper fits a software-engineering or CAiSE-style venue, but the current framing overstates the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real, reproducible engineering contribution with a flawed evaluation and overstated conclusions. The extractor exists, the code and raw outputs are on Zenodo, and running it against 30 live SaaS pages is a legitimate check. Feature extraction picked up most of what was there. That is the part worth keeping.\n\nThe soft spots are exactly where the reader's report puts them. In Section 4.1, a True Negative is defined as data not extracted because it was absent from the scraped information, “although it might be accessible through dynamic interactions.” The rubric gives full credit for those. That means the system is rewarded for not clicking buttons. The paper then says only 15 of 30 sites yielded features, usage limits, and add-ons; the other 15 are silently dropped from those metrics. Every headline accuracy/recall figure in Figure 3 inherits that inflation. The abstract claim about 'high accuracy and recall across all steps' does not survive contact with the definition.\n\nThe other overreach is scope. Section 3.2 says only the Information Extractor was implemented; the Process Engine and Results Modeler are not built or tested. But the abstract and conclusion talk about automating the full transformation 'across all steps.' That is a mismatch.\n\nThe gold standard is the authors' own dataset and schema. That is not fatal, but with the TN rule it makes it hard to tell what should have been in the expected-positive set. There are also no baselines—no comparison with a scraping-only pipeline, a smaller-context LLM, or a cheaper heuristic. So we don't learn much about whether the LLM is doing the work.\n\nNone of these are deep conceptual flaws. The scoring definition can be fixed by separating true negatives from unextracted dynamic content, reporting the 15/15 split, and restricting claims to the implemented component. The artifact is worth refereeing because it is concrete and independent results are cheap to check. I'd send it to peer review with expectations of major revision, and I'd want to see the revised evaluation before citing it. For a reading group, it would be a useful case study in how metric definitions create headline results.","headline":"Useful extractor with released code; evaluation credits unextracted dynamic content as correct, inflating the reported accuracy.","tokens_in":9741,"tokens_out":2080,"would_cite":false,"duration_ms":21706,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM-driven Information Extractor can automate the first step of converting static SaaS pricing pages into machine-readable intelligent-pricing models, with validation on 30 commercial SaaS websites reporting mean…","keywords":["Software as a Service","iPricing","Large Language Model","information extraction","SaaS pricing","pricing automation","Pricing4SaaS"],"falsifier":"Take a new sample of 20 SaaS pricing pages known to contain add-ons in structured tables, run AI4Pricing2Yaml on them, and have an independent annotator score every extracted item without peeking at the tool's output. If add-on precision remains around 60% or accuracy near 50% while the gold standard counts only visible items, the claim that the extractor achieves high accuracy across the pipeline would fail for the weakest component.","tokens_in":8772,"feed_emoji":"🤖","tokens_out":4926,"duration_ms":46247,"temperature":0.7,"pith_summary":"The paper argues that SaaS pricing pages, which are currently maintained by hand and grow exponentially complex, can be automatically turned into intelligent pricing: dynamic, machine-readable models in YAML. It presents AI4Pricing2Yaml, an LLM-driven information extractor that scrapes a pricing URL and uses a large language model to pull out plans, features, usage limits, and add-ons, then reports validation on 30 commercial SaaS sites. The strongest results are for features (88.2% mean accuracy, 96.4% recall) and plans (100% recall), while add-ons and usage limits are far less reliable (53.5% and 67% mean accuracy). The paper frames the work as the first step toward fully automated pricing transformation.","feed_headline":"LLM extractor hits 88% accuracy on SaaS pricing features","feed_subtitle":"A scraping-plus-LLM pipeline automates the first step of intelligent pricing, with feature recall near 96 percent.","key_machinery":"The load-bearing mechanism is the Information Extractor, a pipeline that pairs Selenium, which renders JavaScript-generated HTML, with the Gemini 1.5 Flash LLM, whose large context window lets the model see an entire pricing page at once. Prompt engineering directs the model to output the four pricing elements—plans, features, usage limits, and add-ons—in the structure defined by the Pricing4SaaS metamodel, ready to be serialized with the Pricing2Yaml syntax. The extractor is designed for pages that contain a comparison table with all plans and features; pages that hide information behind clicks or lack structured tables are known failure cases.","core_discovery":"The central claim is that an LLM-based Information Extractor can automate the first stage of transforming a static HTML pricing page into an intelligent pricing expressed with the Pricing4SaaS metamodel, and that the resulting extraction is accurate and complete enough to be feasible. Concretely, the paper reports that on a dataset of 30 commercial SaaS websites (more than 150 price configurations), the system achieves 100% recall for plans, 96.4% recall for features, 77.8% for usage limits, and 81% for add-ons, with precision trailing at 61.4%, 91.1%, 83.8%, and 63% respectively. The authors state that feature extraction performed well, that plans are extracted with perfect recall, and that challenges remain in add-ons and usage limits, as well as in dynamic or complex pages.","pith_inferences":["If the paper's claim of feasibility is accepted for feature extraction, a natural next step would be to treat the extractor as a monitoring sensor that watches a pricing page over time and emits diffs, turning pricing evolution into a versioned software artifact.","The reported accuracy may be optimistic: counting items hidden behind clicks as True Negatives inflates scores, so an independent annotation that treats unseen data as unknown, rather than correct, would likely lower the numbers, especially for add-ons.","Because the pipeline's success is tied to comparison-table layouts, a testable extension would be an LLM agent that clicks through dynamic content; the paper itself suggests this, and it would directly address the 15 sites where the extractor returned nothing."],"forward_implications":["If the extraction is reliable, DevOps teams could stop hand-maintaining pricing models; changes to a pricing page could be reflected in a machine-readable model automatically.","High recall on plans and features means an automated competitive-analysis tool could monitor dozens of competitors' pricing pages for changes.","The remaining gap lies in add-ons and usage limits; improving those requires either better prompts or a separate validation stage, since precision for add-ons is already low.","Since only 15 of 30 sites yielded features, usage limits, and add-ons, the approach's current scope is effectively limited to comparison-table-based pricing pages; broadening it requires handling dynamic content."],"supporting_citations":[{"why":"Supplies the dataset of 30 SaaS pricings used for all reported validation metrics.","marker":"[7]"},{"why":"Defines the Pricing4SaaS metamodel that structures plans, features, usage limits, and add-ons in the output.","marker":"[5]"},{"why":"The Gemini 1.5 Flash model whose large context window lets the extractor process full pricing-page HTML.","marker":"[8]"},{"why":"Defines the Pricing2Yaml output syntax that serializes the extracted model into machine-readable YAML.","marker":"[12]"},{"why":"Documents the exponential growth of SaaS configuration space, motivating the need for automated pricing transformation.","marker":"[4]"}],"fun_headline_variants":["LLM extraction hits 100% plan recall, 96% features on SaaS pricing","AI4Pricing2Yaml: LLM turns HTML pricing into structured data with high recall","LLM automates SaaS pricing extraction: perfect plan recall, gaps in add-ons","Automated intelligent pricing: LLM extracts features at 96% recall","LLM-based tool transforms static SaaS pricing pages into intelligent models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the manually created gold standard for the 30 websites as correct, and counts items that were never extracted (because they sit behind clicks or in unstructured layouts) as True Negatives, which inflates all reported accuracy numbers.","fun_headline_variants_meta":{"raw":{"variants":["LLM extraction hits 100% plan recall, 96% features on SaaS pricing","AI4Pricing2Yaml: LLM turns HTML pricing into structured data with high recall","LLM automates SaaS pricing extraction: perfect plan recall, gaps in add-ons","Automated intelligent pricing: LLM extracts features at 96% recall","LLM-based tool transforms static SaaS pricing pages into intelligent models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1618,"prompt_tokens":1001,"completion_tokens":617,"prompt_tokens_details":{"cached_tokens":896},"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":105,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":105,"tokens_out":617,"duration_ms":301239,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":896,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:53:18.286492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new sample of 20 SaaS pricing pages known to contain add-ons in structured tables, run AI4Pricing2Yaml on them, and have an independent annotator score every extracted item without peeking at the tool's output. If add-on precision remains around 60% or accuracy near 50% while the gold standard counts only visible items, the claim that the extractor achieves high accuracy across the pipeline would fail for the weakest component.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 30 SaaS pricings used for all reported validation metrics."},{"cited_title":"In: Intelligent Information Systems","cited_arxiv_id":null,"evidence_quote":"Defines the Pricing4SaaS metamodel that structures plans, features, usage limits, and add-ons in the output."},{"cited_title":"https://pricing4saas-docs.vercel.app/docs/2.0","cited_arxiv_id":null,"evidence_quote":"Defines the Pricing2Yaml output syntax that serializes the extracted model into machine-readable YAML."},{"cited_title":"In: Service- Oriented Computing","cited_arxiv_id":null,"evidence_quote":"Documents the exponential growth of SaaS configuration space, motivating the need for automated pricing transformation."}],"review_version":1}