{"id":"47e4a7ca-9ac1-4af5-a0bb-cb4a00272a8e","arxiv_id":"2412.10717","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A lightweight Java/Swing n-gram language modeling platform with smoothing and dynamic corpus management, whose efficiency and accuracy claims are under-supported and internally inconsistent.","lead":"HITgram is a Java-based platform for building and experimenting with n-gram language models on an ordinary personal computer. It targets students and developers who want a lightweight, interpretable alternative to large language models, but its reported performance and accuracy claims are not backed by the experiments shown.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline scaling claims (320MB in 62s; 1GB in under 298s) are unsupported and appear to contradict Table 2's measured 13MB timings; the central efficiency claim must be verified at the stated scale.","rationale":"The reader's weakest assumption is that Table 2's small-corpus timings extrapolate to the 320MB and 1GB claims, and that the observed scaling is linear. My stress test confirms this is the most load-bearing point and sharpens it: the paper's own Table 2 data are not merely missing large-scale runs; they are in tension with the abstract's numbers. A linear extrapolation of the last Table 2 row gives roughly 5.6s for n=2 at 320MB (or about 102s including tokenization), not 62s, and roughly 344s total for 1GB n=4, not under 298s. The table also contains an apparent non-linearity in the final two rows that contradicts the paper's stated linear-scaling interpretation. The perplexity evaluation is also weak, but the efficiency numbers are the primary claimed contribution, so the scaling discrepancy is the single most decisive issue. Since the reader already recommended REJECT, my analysis does not change the verdict; it strengthens it by identifying an internal consistency problem rather than relying only on missing evidence. I would keep the verdict as REJECT on the grounds that the headline performance claims are unverified and contradicted by the reported data.","tokens_in":182,"tokens_out":3264,"duration_ms":44233,"concrete_test":"Run the public HITgram repository (https://github.com/chandan789maity/HITgram) on a machine matching the stated 8GB RAM configuration, and measure load+tokenize plus build time on (a) the 13.27MB corpus from Table 2, (b) a 320MB corpus, and (c) a 1GB corpus, for n=2 and n=4. Check whether the 13.27MB row reproduces and whether the 320MB and 1GB totals match the abstract's 62s and <298s claims; also check whether scaling from the reproduced 13MB measurement is linear within, say, 20%. If the advertised times are not reproduced, the central efficiency claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that HITgram is fast and memory-light at the scales advertised in the abstract: 2-grams from 320MB in 62 seconds and 4-grams from 1GB in under 298 seconds. The only reported build timings (Table 2) stop at 13.27MB, so the abstract numbers are never directly measured. More seriously, the numbers in Table 2 are not consistent with the abstract under any simple scaling. The last row of Table 2 reports a 13,273.6 KB corpus with load+tokenize time 4001.5 ms, n=2 build time 231.5 ms, and n=4 build time 450.6 ms. Scaling the n=2 build linearly to 320MB gives roughly 5.6 seconds, not 62 seconds. If load+tokenize is included, the linear projection is about 102 seconds, still not 62 seconds. For the 1GB claim, scaling the n=4 row linearly gives about 35 seconds of build plus about 309 seconds of load+tokenize, roughly 344 seconds total, not under 298 seconds. The table also shows a non-linear jump: from 11.92MB to 13.27MB (about +11% in size), the n=4 build time rises from 121.5 ms to 450.6 ms, a 3.7x increase, while load+tokenize rises from 2779.3 ms to 4001.5 ms. The text itself claims scaling is linear, but the reported data do not support that. Because the efficiency claims are the stated contribution, this internal inconsistency is load-bearing: if the abstract's numbers cannot be reproduced or explained by a concrete protocol, the platform's core value proposition is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HITgram is presented as a lightweight Java-based platform for building and experimenting with n-gram language models (unigrams through 4-grams), featuring customizable tokenization, Laplace/Add-k/Good-Turing smoothing, context-sensitive weighting, and dynamic corpus management. The paper claims high efficiency — 50,000 tokens/second, 2-gram construction from a 320 MB corpus in 62 seconds, and 4-gram construction from a 1 GB file in under 298 seconds on an 8 GB RAM system — and evaluates the platform with perplexity numbers and build-time tables on much smaller corpora. The paper is written as a system description with a GUI screenshot and code snippets, but the experimental validation is not sufficient to support the central efficiency and accuracy claims.","tokens_in":9257,"tokens_out":5338,"duration_ms":46152,"significance":"If the headline efficiency numbers were reproducible, HITgram could be a useful pedagogical and lightweight tool for n-gram experimentation in resource-constrained settings, and the open-source release would be a practical asset. The paper also correctly emphasizes interpretability and low computational cost relative to large language models. However, the significance as stated is not established: the reported experiments stop at 13 MB corpora, the scaling behavior in Table 2 contradicts the claimed linear trend and the abstract's 320 MB/1 GB figures, and the perplexity evaluation is performed on a single sentence with no held-out protocol. The paper's own Table 1 footnote ('results may vary depending on machine configuration') and Table 2 footnote ('measured using AMD Ryzen mid-range performance processors') acknowledge environmental sensitivity but do not mitigate the internal inconsistencies.","major_comments":[{"comment":"The central efficiency claims are not supported by any reported measurement. Table 2 stops at 13,273.6 KB, while the abstract claims 2-gram construction from a 320 MB corpus in 62 seconds and 4-gram construction from a 1 GB file in under 298 seconds; neither figure appears in the table or elsewhere. Linear extrapolation from the last row of Table 2 gives roughly 96.5 s of load+tokenize plus 5.6 s of n=2 build time for 320 MB (total about 102 s, not 62 s), and roughly 302 s of load+tokenize plus 34 s of n=4 build time for 1 GB (total about 335 s, not under 298 s). The abstract's numbers are therefore unsupported and inconsistent with the paper's own data.","section":"Abstract; §5, Table 2"},{"comment":"The text states that 'n-gram model construction time increases linearly with corpus size and n', but Table 2 contradicts this. From 10,302.58 KB to 11,922.73 KB, a 15.7% size increase, the n=3 build time rises from 31.7 ms to 109.7 ms (about 3.5x), and from 11,922.73 KB to 13,273.6 KB, an 11.3% increase, the n=4 build time rises from 121.5 ms to 450.6 ms (about 3.7x). The n=2 build time also jumps non-monotonically from 32.2 ms at 10,302.58 KB to 77.4 ms at 11,922.73 KB, and then to 286.5 ms at 13,051.37 KB before falling to 231.5 ms at 13,273.6 KB. These data do not support the linear-scaling claim and invalidate the extrapolation to the abstract's large-corpus figures.","section":"§5, Table 2 and text"},{"comment":"The perplexity evaluation is not a valid assessment of model quality. The text says 'The analysis is based on the sentence: \"this is a\"', and no held-out split or training/test separation is described. If that sentence appears in the training corpus, the perplexity numbers in Table 1 are a self-consistency check, not a measure of generalization to unseen text. The caption 'Perplexity values of test set of words' does not specify a held-out protocol, so the accuracy and smoothing claims are not substantiated.","section":"§5, Table 1 and 'Dataset Description'"},{"comment":"The Good-Turing formula as written, P(wi|wi−1) = N1/N, is incorrect. Good-Turing smoothing estimates probability mass for unseen events using counts of counts (e.g., N_r, the number of n-grams appearing r times) and is not a conditional bigram probability of this form. Since Good-Turing smoothing is advertised as a core platform feature, this technical error undermines the description of the method and could mislead users of the platform.","section":"§4, 'Good-Turing Smoothing'"}],"minor_comments":[{"comment":"The perplexity formula is garbled: 'PP(W) = N p P(w1, w2, . . . , wN)' does not render as the N-th root of the inverse probability product. Please typeset it as PP(W) = (∏_{i=1}^N 1/P(wi | w1,...,w_{i−1}))^{1/N}.","section":"§5, perplexity formula"},{"comment":"There is a typo 'e-hance' in the abstract; it should be 'enhance'.","section":"Abstract"},{"comment":"The platform name is inconsistently written as 'HITgram' and 'HITGram' (e.g., Section 4 heading); please standardize.","section":"Throughout"},{"comment":"The sentence 'efficiently processing PDFs (e.g., a 0.8 GB file to 10,302 KB in 2,689 ms)' is unclear: is 0.8 GB the PDF file size and 10,302 KB the extracted text size? If so, state this explicitly, and clarify why the extracted text is much smaller than the original file.","section":"§5, 'Experimental Setup'"},{"comment":"The GitHub repository is cited without a commit hash or version tag, and the dataset URLs are only given generically (Kaggle and textfiles.com); please provide stable identifiers or direct links so the experiments can be reproduced.","section":"§5, 'Reproducibility'"},{"comment":"Several references are incomplete or seemingly unrelated: [4], [6], and [10] truncate author lists with 'et al.' without standard formatting, and [5] ('Mesh Generation - A Survey') does not appear relevant to Laplace smoothing as cited. Please verify and complete the bibliography.","section":"References"},{"comment":"Figure 1 plots single measurements without error bars or repeated runs; given the Table 2 footnotes about machine dependence, it would be helpful to report the number of runs and variance.","section":"§5, Figure 1"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a preliminary system/project report. The central quantitative claims are not merely under-supported; they are contradicted by the paper's own Table 2, and the perplexity evaluation is methodologically invalid. These issues are load-bearing for the stated contribution, and fixing them would require rerunning the evaluation at the advertised corpus sizes with a proper held-out protocol, which is beyond a normal revision. In addition, the novelty relative to established n-gram toolkits (e.g., KenLM, SRILM) is not discussed or benchmarked. I would therefore recommend rejection rather than major revision, despite the availability of source code and the paper's clear organization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: HITgram is a decent little Java/Swing tool for teaching n-gram models, but the paper's central efficiency claims are not backed by the evidence it reports. The 320MB/62s and 1GB/298s figures from the abstract appear nowhere in the tables. Extrapolating the last row of Table 2 linearly, a 320MB bigram build should take about 5.6 seconds, not 62; including load and tokenization, roughly 102 seconds. The 1GB claim also misses: about 352 seconds total, not under 298. Worse, the table itself contradicts the paper's asserted linear scaling: from 11.9MB to 13.3MB, the n=4 build time jumps from 121.5ms to 450.6ms, a 3.7x increase for an 11% size increase. So the headline numbers are either wrong or missing.\n\nWhat is genuinely here: a clean implementation of standard n-gram modeling with Laplace, Add-k, and Good-Turing smoothing, a simple log-frequency weighting, dynamic corpus updates, and pruning. For an undergraduate lab or a quick demo, this could have value. The GitHub repository with an installer is a real artifact, and the citations to textbook material are appropriate.\n\nThe soft spots: the perplexity evaluation is essentially meaningless. The paper says the analysis is based on the sentence \"this is a\" — one sentence, no held-out protocol, no baseline comparison, no error bars. The \"context-sensitive weighting\" is just log(1+frequency), a standard frequency transform, not a novel contribution. And there is no comparison with existing n-gram toolkits such as KenLM or SRILM, so the efficiency claims have no calibration.\n\nThese are load-bearing flaws because the stated contribution is efficiency and accessibility. Without valid timing data at the claimed scales and a proper held-out evaluation, the paper is a tool description, not a research result. It could be revised into a modest systems/education paper, but not as it stands.\n\nWho is this for? Educators and students wanting a hands-on n-gram demo. Not for researchers. My recommendation: desk-reject or send back for major revision with a demand for real measurements at 320MB and 1GB, a proper perplexity evaluation, and a comparison to existing toolkits. A serious referee could check the arithmetic and likely arrive at the same verdict.","headline":"The abstract's efficiency numbers are unsupported by the paper's own Table 2, and the perplexity evaluation is a single sentence — a useful demo tool, not a research result.","tokens_in":9880,"tokens_out":4998,"would_cite":false,"duration_ms":44270,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HITgram is a lightweight n-gram language modeling platform that claims tokenization at 50,000 tokens per second and construction of 4-grams from a 1 GB corpus in under 298 seconds on an 8 GB RAM machine.","keywords":["n-gram model","language modeling","Laplace smoothing","Good-Turing smoothing","resource-constrained NLP","perplexity","text generation","lightweight platform"],"falsifier":"Run HITgram on a 320 MB plain-text corpus on an 8 GB RAM machine and measure the time to build a bigram model; if the build takes substantially longer than 62 seconds, or if tokenization throughput falls well below 50,000 tokens per second, the paper's central efficiency claim fails. A second check: compute perplexity of a trigram model on a held-out corpus of at least a few hundred sentences; if it does not beat a uniform baseline, the claimed accuracy advantage is not supported.","tokens_in":8707,"feed_emoji":"⚡","tokens_out":6107,"duration_ms":46159,"temperature":0.7,"pith_summary":"HITgram is a lightweight, GUI-based platform for building and experimenting with n-gram language models, targeting users with limited computational resources. The paper argues that classical n-gram models remain practical alternatives to large neural language models, and presents HITgram as a way to build, smooth, and query such models on a standard laptop. The central evidence is empirical: tokenization at up to 50,000 tokens per second, construction of 2-grams from a 320 MB corpus in about 62 seconds, and construction of 4-grams from a 1 GB file in under 298 seconds on an 8 GB RAM machine. These numbers support the claim that n-gram modeling can be done efficiently and accessibly without specialized hardware.","feed_headline":"N-gram platform: 50,000 tokens/sec on 8 GB RAM","feed_subtitle":"Classical n-gram modeling on a laptop, with smoothing for unseen word sequences and 1 GB corpora buildable in minutes.","key_machinery":"The central mechanism is the n-gram frequency map, a key-value structure that stores the preceding $n-1$ words as the key and a map of next-word counts as the value. Model construction updates this map using putIfAbsent and merge operations; prediction reads the map and applies Laplace smoothing, $P(w_i|w_{i-1}) = \\frac{\\text{Count}(w_i, w_{i-1})+1}{\\text{Count}(w_{i-1})+V}$, to guarantee non-zero probabilities for unseen sequences. A logarithmic transformation $\\log(1+\\text{frequency})$ stabilizes weights for infrequent n-grams, and dynamic corpus management allows new text to be merged into an existing model for incremental learning.","core_discovery":"The paper's central claim is that a well-engineered n-gram platform can make classical language modeling practical in resource-constrained environments. HITgram implements n-gram models from unigrams to 4-grams using a key-value store for context-to-next-word counts, and augments them with Laplace, Add-k, and Good-Turing smoothing plus a logarithmic context-frequency weighting to keep rare sequences from getting zero probability. On the paper's own measurements, the platform tokenizes English text at up to 50,000 tokens per second, builds 2-grams from a 320 MB corpus in 62 seconds, and constructs 4-grams from a 1 GB file in under 298 seconds on an 8 GB RAM system. The paper interprets these results as evidence that n-gram models remain a viable, interpretable alternative to LLMs for tasks such as autocomplete, predictive text, and speech-to-text preprocessing.","pith_inferences":["The paper's efficiency claims are extrapolated from corpora of at most 13 MB; actually reproducing the 320 MB and 1 GB figures on the same hardware would settle whether the linear scaling assumption is correct.","The reported perplexity values are computed from the single test sentence \"this is a\", so they are not a reliable measure of model quality; a proper held-out perplexity evaluation would be a natural next step for the platform.","If the scaling results do reproduce, HITgram-style n-gram models could serve as low-cost, interpretable baselines in hybrid systems that combine n-gram backoff with neural components.","The logarithmic weighting is presented as a stabilization heuristic; its effect on predictive accuracy for rare n-grams is a testable extension that the paper does not evaluate."],"forward_implications":["If the reported throughput holds, n-gram models can be trained and queried in real time on standard laptops, making autocomplete and predictive text feasible on low-end hardware.","The 1 GB corpus build time under 298 seconds on 8 GB RAM suggests that classical n-gram methods scale to moderately large datasets without GPUs, an option for small-scale NLP deployments.","The smoothing options (Laplace, Add-k, Good-Turing) and context-sensitive weighting address data sparsity in a practical platform, allowing unseen sequences to receive non-zero probabilities.","The dynamic corpus management feature supports incremental learning, so models can be updated as new text arrives without a full rebuild.","HITgram provides a concrete baseline for comparing lightweight n-gram models against small language models in resource-constrained educational and accessibility settings."],"supporting_citations":[{"why":"Supplies the standard n-gram formulation and perplexity evaluation used by the platform.","marker":"[9]"},{"why":"Quantifies LLM memory demands (about 140 GB VRAM for LLaMA-70B), grounding the resource-constraint motivation.","marker":"[21]"},{"why":"Provides the Kneser-Ney smoothing baseline that HITgram's smoothing options complement.","marker":"[13]"},{"why":"Defines backoff-interpolation estimation, the conceptual basis for combining lower-order n-grams in the platform.","marker":"[12]"},{"why":"Identifies the data sparsity problem that motivates HITgram's smoothing and weighting features.","marker":"[2]"},{"why":"Introduces Laplace smoothing, the method HITgram applies to guarantee non-zero probabilities.","marker":"[5]"}],"fun_headline_variants":["HITgram: 50k tokens/sec n-gram modeling on 8GB RAM","N-gram platform: 2-grams from 320MB in 62 seconds","Build 4-grams from a 1GB corpus in under 5 minutes","HITgram: efficient n-gram experiments without the GPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that build times measured on small corpora (up to 13 MB) scale linearly to the 320 MB and 1 GB timings claimed, and that perplexity computed on the single phrase \"this is a\" is a meaningful measure of prediction quality.","fun_headline_variants_meta":{"raw":{"variants":["HITgram: 50k tokens/sec n-gram modeling on 8GB RAM","N-gram platform: 2-grams from 320MB in 62 seconds","Build 4-grams from a 1GB corpus in under 5 minutes","HITgram: efficient n-gram experiments without the GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2611,"prompt_tokens":871,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":1654}},"tokens_in":487,"tokens_out":1740,"duration_ms":13940,"temperature":1.0,"reasoning_tokens":1654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:40:21.555470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HITgram on a 320 MB plain-text corpus on an 8 GB RAM machine and measure the time to build a bigram model; if the build takes substantially longer than 62 seconds, or if tokenization throughput falls well below 50,000 tokens per second, the paper's central efficiency claim fails. A second check: compute perplexity of a trigram model on a held-out corpus of at least a few hundred sentences; if it does not beat a uniform baseline, the claimed accuracy advantage is not supported.","supporting_citations":[{"cited_title":"to Natural Language Processing, Computational Linguistics, and Speech Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the standard n-gram formulation and perplexity evaluation used by the platform."},{"cited_title":"In: 1995 international conference on acoustics, speech, and signal processing","cited_arxiv_id":null,"evidence_quote":"Provides the Kneser-Ney smoothing baseline that HITgram's smoothing options complement."},{"cited_title":"IEEE transactions on acoustics, speech, and signal processing 35(3), 400–401 (1987)","cited_arxiv_id":null,"evidence_quote":"Defines backoff-interpolation estimation, the conceptual basis for combining lower-order n-grams in the platform."},{"cited_title":"In: Text, Speech and Dialogue: 9th International Conference, TSD 2006, Brno, Czech Republic, Sep 11-15, 2006","cited_arxiv_id":null,"evidence_quote":"Identifies the data sparsity problem that motivates HITgram's smoothing and weighting features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Laplace smoothing, the method HITgram applies to guarantee non-zero probabilities."}],"review_version":1}