{"id":"054d791a-8757-46be-b78e-a16691bb9b91","arxiv_id":"2508.14718","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-2 large beats LSTM/RNN baselines on recipe generation metrics, but the paper's tokenization contribution lacks any ablation support.","lead":"This paper fine-tunes GPT-2 models for recipe generation and compares them with LSTM/RNN baselines on 51,000 recipes, reporting large wins for GPT-2. The claimed key contribution, a custom fraction-aware tokenizer, is never tested on its own, and no code is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the custom tokenizer: the key contribution claim is confounded with model scale and architecture.","rationale":"The reader's weakest assumption is precisely that the custom tokenization is never tested in isolation, making the contribution claim unsupported. My analysis confirms this: the comparison between GPT-2 (custom tokenizer) and LSTM/RNN (word-level tokenizer) confounds tokenization with model scale, pretraining, and decoding strategy. The additional observation about the abstract's misstated BERTScore relative improvement (0.72 is the RNN baseline, not the best recurrent baseline of 0.87) strengthens the case that the headline numbers are misleading. However, the paper's core comparative benchmark, while flawed in its attribution, still provides a potentially useful reference point if the authors release code and add the missing ablations. Therefore the conditional verdict remains appropriate; no change is needed.","tokens_in":8520,"tokens_out":3208,"duration_ms":33395,"concrete_test":"Fine-tune GPT-2 large on the same 5-cuisine train split with the default GPT-2 BPE tokenizer, adding the structural markers such as <INGR_START> as regular vocabulary entries (no fraction-preservation rules), and evaluate on the identical test set with the same metrics and decoding hyperparameters. Then compare to the Table 4 GPT-2 large row. If BERTScore F1 and perplexity are within run-to-run variance (say, ΔF1 ≤ 0.01, ΔPPL ≤ 1), the tokenization contribution is not empirically supported. This directly tests whether Section 3.2.2's tokenizer is the driver of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a targeted tokenization strategy (23 fraction tokens + structural markers) is a key contribution and that GPT-2 large with this tokenizer sets a new SOTA. However, the experimental design never isolates the tokenizer. GPT-2 models are trained with the custom tokenizer (§3.2.2), while the LSTM/RNN baselines use a word-level tokenizer (§3.1). Because architecture, parameter count, pretraining, and tokenization all vary between the compared systems, the observed metric differences (e.g., BERTScore 0.92 vs 0.87 for LSTM) cannot be attributed to the tokenization. Section 5.2 asserts the custom tokens 'proved highly effective' without any ablation where tokenizer is the only changed variable. Additionally, the abstract's headline '>20% relative BERTScore improvement' compares 0.92 to the RNN baseline 0.72, not the best recurrent baseline (LSTM, 0.87); the actual relative improvement over the best recurrent baseline is ~5.7%. This misstatement further obscures the contribution's magnitude. The load-bearing assumption—that the tokenization is responsible for the gains—is untested and hence unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares fine-tuned GPT-2 small and large models against word-level LSTM and RNN baselines for recipe generation on the 5-cuisine RecipeDB corpus. The authors state their key contribution as a domain-specific tokenization strategy that adds 23 fraction tokens and custom structural markers, and they report that GPT-2 large achieves the best automatic metric scores, including a BERTScore F1 of 0.92 and perplexity of 20.67. The paper also argues that this tokenization strategy is responsible for the improvements and claims a new state of the art for the benchmark.","tokens_in":8813,"tokens_out":3697,"duration_ms":40934,"significance":"If the central claim were supported, the paper would provide a useful benchmark and a practical tokenization recipe for structured, number-heavy generation tasks. The release of code, the use of seven automatic metrics, and the comparison across two GPT-2 scales and recurrent baselines are positive elements. However, the central claim that the custom tokenization is responsible for the gains is not supported by the experimental design: the tokenizer is varied together with architecture, scale, and pretraining, and no ablation isolates it. The paper also contains a quantitative misstatement in the abstract and a vocabulary-size inconsistency that casts doubt on the implementation details. The benchmark itself is useful, but the headline claims need substantial revision and additional experiments.","major_comments":[{"comment":"The paper's key contribution is the targeted tokenization strategy, but no experiment isolates its effect. GPT-2 models use the custom tokenizer introduced in §3.2.2, while the LSTM/RNN baselines use a word-level tokenizer (§3.1). Thus the better scores of GPT-2 large in Table 4 could be due to architecture, scale, pretraining, or tokenization. Section 5.2 asserts that the custom tokens 'proved highly effective' with no control condition, e.g., GPT-2 large with and without the augmented fractions and structural markers. Without such an ablation, the central claim is not established.","section":"§3.2.2, §5.2, Table 4"},{"comment":"The abstract claims a '>20% relative improvement in BERTScore (F1) (0.92 vs 0.72) over the best recurrent baseline.' In Table 4 the 0.92 vs 0.72 comparison is against the RNN baseline, not the best recurrent baseline; the LSTM baseline has BERTScore 0.87, so the relative improvement is (0.92−0.87)/0.87 ≈ 5.7%, not over 20%. The paper also states that GPT-2 large 'sets a new state-of-the-art' for the 5-cuisine benchmark, but Table 1 reports prior RecipeGPT and Ratatouille results without a direct, same-protocol comparison on this benchmark. The SOTA claim is therefore unsupported.","section":"Abstract, Table 4, Table 1"},{"comment":"Perplexity is reported for models with different tokenizers: GPT-2 uses a byte-level BPE augmented with custom tokens, while the recurrent baselines use a word-level vocabulary. Perplexity is not comparable across different tokenizations because the probability mass is distributed over different token units. Therefore the '69.8% perplexity reduction' claimed in the abstract is not a meaningful head-to-head comparison. A valid comparison would require either a shared tokenizer or a text-level perplexity estimate.","section":"§4.2, Table 4"},{"comment":"Table 3 reports the vocabulary size as '∼50,257' (Base + Special). This is exactly the standard GPT-2 vocabulary size (50,257). If the paper really added 23 fraction tokens, custom structural markers, and a [PAD] token, the vocabulary size should be larger. Either the tokens were not actually added, the table is wrong, or the base vocabulary was replaced. This inconsistency directly affects the credibility of the tokenization contribution and needs to be resolved with the actual tokenizer configuration.","section":"Table 3, §3.2.2"},{"comment":"All results in Table 4 are point estimates with no measures of variance. There is no report of multiple random seeds, no error bars, and no significance testing. Given that the differences between GPT-2 small and large are small (e.g., BERTScore 0.90 vs 0.92), and the sampling-based generation in §3.2.4 introduces stochasticity, the results may not be stable. The paper should report multiple seeds with standard deviations or confidence intervals, and use appropriate significance tests for the headline comparisons.","section":"§4.1, Table 4"}],"minor_comments":[{"comment":"Table 1 lists BLEU-4 values of 0.085 and 0.080 with the column header 'BLEU-4 (%)'. If these are percentages, they are incompatible with Table 4, where BLEU-4 values are reported as 0.08–0.26. Clarify the unit convention consistently.","section":"Table 1"},{"comment":"The text says 'the ’gpt2(774 Million)’ variant'; this should be 'GPT-2 large' or 'gpt2-large' for consistency with the rest of the paper. Also check the formatting in the same sentence.","section":"§3.2.3"},{"comment":"'V ocabulary' is a typo for 'Vocabulary'.","section":"Table 3"},{"comment":"The sentence 'as noted by reviewers' in the future-work paragraph is inappropriate for a submitted manuscript; remove or rephrase.","section":"§6"},{"comment":"The interpretation of BERTScore as 'capturing underlying cooking concepts' is overstated; BERTScore measures embedding similarity and should not be equated with conceptual correctness. Consider softening the claim.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark component, but the main claimed contribution (tokenization) is confounded with scale and architecture. The vocabulary-size inconsistency in Table 3 is a red flag that the tokenization implementation may not match the description. I would not recommend rejection because the issue is fixable with an ablation and corrected reporting, but the current version's central claim is not supported. If the authors cannot add an ablation, they should re-scope the claims away from 'key contribution' and 'state of the art.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a fine-tuning comparison for recipe generation on the 5-cuisine corpus. The headline numbers are plausible: GPT-2 large beats GPT-2 small and the LSTM/RNN baselines on all seven metrics. What's actually new is the fraction-aware tokenizer with 23 fraction tokens and structural markers. That idea is sensible for a domain where \"1/2 cup\" and section boundaries matter. The paper is also honest in its limitations section: it flags factual inaccuracy, ingredient hallucination, repetition, and the need for human evaluation. Credit for that.\n\nThe soft spots are structural. The tokenizer is never ablated. All GPT-2 runs use the custom tokenizer, and all recurrent baselines use a word-level tokenizer. So you cannot attribute the gains to tokenization; it is confounded with architecture, scale, and pretraining. Section 5.2 says the custom tokens \"proved highly effective,\" but no experiment isolates the tokenizer as the only variable. A simple ablation—GPT-2 with and without fraction tokens, or the custom tokenizer applied to the LSTM—would fix this.\n\nThere is also a factual error in the abstract: the \">20% relative improvement in BERTScore (0.92 vs 0.72) over the best recurrent baseline\" is wrong. 0.72 is the RNN baseline; the LSTM is 0.87. The actual relative improvement over the best recurrent baseline is about 5.7%. That misstates the contribution's magnitude. Also no error bars, no seeds, no significance tests, so even the corrected gap may be noise. The paper says code is on GitHub, but I don't see a link; without code and data, reproducibility suffers.\n\nThe tokenization idea is worth taking seriously—it could transfer to other structured text with quantities. But as presented, the central claim is unsupported. The benchmark itself is routine: RecipeGPT and Ratatouille already fine-tuned GPT-2 for recipes. The paper does add a standardized evaluation on the 5-cuisine corpus with seven metrics, which would be more useful if the code and data were released. One oddity: the future work mentions \"as noted by reviewers,\" which is an odd thing to have in an arXiv preprint, but it is not a substantive flaw.\n\nWho is this for? People working on recipe generation or domain-specific tokenization. It could be a workshop or short-paper submission after major revision. It deserves peer review because the claim is testable and an ablation could make it acceptable. My recommendation: send it out, but the referee should demand an ablation and a corrected abstract. If the authors cannot show the tokenizer matters, the paper reduces to a routine benchmark.","headline":"A standard GPT-2-vs-recurrent benchmark whose stated key contribution, fraction-aware tokenization, is never isolated; the abstract also overstates the gain by comparing against the wrong baseline.","tokens_in":9264,"tokens_out":1890,"would_cite":false,"duration_ms":19886,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning GPT-2 large with a fraction-aware tokenizer outperforms recurrent baselines on a 51,000-recipe benchmark.","keywords":["recipe generation","GPT-2 fine-tuning","domain-specific tokenization","fraction tokens","BERTScore","natural language generation","RecipeDB","benchmark"],"falsifier":"Fine-tune GPT-2 large on the same 5-cuisine split using the stock GPT-2 tokenizer with identical data, sampling, and training settings; if BERTScore F1 and perplexity remain near 0.92 and 20.67, the custom tokenization is not the cause. A more direct test would measure exact reproduction rates of fraction tokens in generated text.","tokens_in":8430,"feed_emoji":"🍳","tokens_out":3278,"duration_ms":38512,"temperature":0.7,"pith_summary":"This paper tries to establish that fine-tuning a large pre-trained transformer, GPT-2 large (774M parameters), with a domain-specific tokenizer produces better recipe text than traditional LSTM/RNN baselines. The authors build a reproducible benchmark on the 5-cuisine RecipeDB corpus, comparing two GPT-2 sizes against word-level recurrent models across seven automatic metrics. Their headline result is a BERTScore F1 of 0.92 versus 0.72 for the best recurrent baseline, with a 69.8% reduction in perplexity. The paper argues that preserving fraction tokens and structural boundary markers is a key part of this success, while also acknowledging that factual accuracy in generated recipes remains unsolved.","feed_headline":"GPT-2 large tops recipe benchmark with fraction-aware tokens","feed_subtitle":"A 774M-parameter model scored 0.92 BERTScore, 20% above the best recurrent baseline, on 51,000 recipes.","key_machinery":"The central mechanism is the custom tokenizer: GPT-2's byte-level BPE vocabulary is augmented with 23 fraction tokens and structural boundary tokens such as <INGR_START>, <NEXT_INGR>, and <INSTR_END>, and the model's embedding matrix is resized to accommodate them. This preserves exact fractional quantities and keeps recipe sections as single semantic units, which is what the paper claims improves numerical fidelity and structural coherence in generation.","core_discovery":"On the 5-cuisine benchmark, a fine-tuned GPT-2 large model with an augmented tokenizer achieves the best reported recipe generation quality: BERTScore F1 0.92, BLEU-4 0.26, ROUGE-L F1 0.54, METEOR 0.47, diversity 0.73, and perplexity 20.67, beating GPT-2 small and both LSTM and RNN baselines on every metric. The authors attribute this to model scale plus a tokenization scheme that adds 23 common fraction tokens and custom structural markers, preventing numerical quantities and recipe-section boundaries from being fragmented by byte-level BPE. They frame this as a new state-of-the-art for the 5-cuisine recipe generation task.","pith_inferences":["The paper never tests the tokenizer in isolation: since all GPT-2 runs use the custom tokens and all recurrent baselines use standard word-level tokens, the reported gains could come mostly from model scale and architecture rather than fraction-aware tokenization.","A direct, testable extension would be to measure exact fraction reproduction (e.g., what fraction of '1/2', '1/3', '2/3' tokens appear correctly in generated text) rather than relying on aggregate BERTScore.","If tokenization is the true driver, applying the same augmented tokenizer to the LSTM/RNN baselines should narrow the gap substantially; if it does not, the paper's stated contribution is not load-bearing.","The benchmark's usefulness would increase if future work reports variance across multiple random seeds, since the current table does not indicate whether the margins between GPT-2 small and large are stable."],"forward_implications":["Domain-specific tokenization becomes a cheap, reusable lever for structured generation tasks where numbers and format matter.","A 774M-parameter GPT-2 model is sufficient to set a strong benchmark for recipe generation, suggesting that larger general-purpose LLMs are not required for this task.","The seven-metric evaluation suite gives later work a standard way to compare fluency, coherence, semantic relevance, and diversity in recipe generation.","The demonstrated weaknesses in factual accuracy point directly to retrieval-augmented generation and constrained decoding as the next necessary steps.","A fine-tuned text-generation engine of this kind can serve as the backbone for image-to-recipe and recipe-to-image multi-modal systems."],"supporting_citations":[{"why":"Supplies the 5cuisine RecipeDB corpus of about 51,000 recipes that all models are trained and evaluated on.","marker":"Batra et al., 2020"},{"why":"Defines the GPT-2 architectures (small and large) that the paper fine-tunes for recipe generation.","marker":"Radford et al., 2019"},{"why":"RecipeGPT, an earlier GPT-2 recipe generator whose reported BLEU-4 the study extends and compares against.","marker":"Lee et al., 2020"},{"why":"Ratatouille, another GPT-2-based recipe tool, establishes the prior state of the art the paper builds on.","marker":"Goel et al., 2022"},{"why":"A recent comparative benchmark for recipe generation that this study positions itself against and distinguishes itself from.","marker":"Vij et al., 2025"},{"why":"Defines BERTScore, the semantic similarity metric behind the headline 0.92 vs 0.72 result.","marker":"Zhang et al., 2020"},{"why":"Provides the Hugging Face Transformers library used to load, tokenize, and fine-tune GPT-2.","marker":"Wolf et al., 2020"},{"why":"Introduces the Transformer architecture with self-attention, the foundation for the GPT-2 models being fine-tuned.","marker":"Vaswani et al., 2017"}],"fun_headline_variants":["GPT-2 large beats RNN/LSTM in recipe generation with fraction tokens","Fraction tokens give GPT-2 large 20% higher BERTScore on recipe benchmark","GPT-2 large with fraction-aware tokenizer cuts perplexity 70% on recipes","Custom fraction tokens help GPT-2 large top recipe generation benchmark"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper assumes the custom fraction tokens and structural markers are what drive the quality gains, but it never tests that in isolation: every GPT-2 model uses them and every recurrent baseline does not, so tokenization is entangled with model scale and architecture.","fun_headline_variants_meta":{"raw":{"variants":["GPT-2 large beats RNN/LSTM in recipe generation with fraction tokens","Fraction tokens give GPT-2 large 20% higher BERTScore on recipe benchmark","GPT-2 large with fraction-aware tokenizer cuts perplexity 70% on recipes","Custom fraction tokens help GPT-2 large top recipe generation benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001409,"raw_usage":{"total_tokens":5549,"prompt_tokens":782,"completion_tokens":4767,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":4683}},"tokens_in":526,"tokens_out":4767,"duration_ms":38653,"temperature":1.0,"reasoning_tokens":4683,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:18:35.795854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune GPT-2 large on the same 5-cuisine split using the stock GPT-2 tokenizer with identical data, sampling, and training settings; if BERTScore F1 and perplexity remain near 0.92 and 20.67, the custom tokenization is not the cause. A more direct test would measure exact reproduction rates of fraction tokens in generated text.","supporting_citations":[],"review_version":1}