{"id":"0cc821ef-6f9b-4d3d-a396-29f538f55cdf","arxiv_id":"2412.15304","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Small custom GPT-2 models (30-124M parameters) can match larger LLMs on three sensor-classification tasks and run faster on edge computers.","lead":"TinyLLM is a framework for training very small language models (30-124 million parameters) on custom data and running them on edge computers for sensor-based tasks like gesture and location recognition. The paper's experiments suggest these tiny models can match much larger models on these tasks while generating tokens far faster on single-board computers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The first-3-4-token accuracy check in Fig. 10 may penalize large models for verbose correct outputs, inflating the apparent superiority of small custom models; this must be tested before accepting the 'surpass' claim.","rationale":"The reader's weakest assumption focuses on representativeness of the three evaluation datasets. That is a legitimate external-validity concern: two of the three datasets are in-house and all are tiny. However, even if the datasets were perfectly representative, the central claim that small models 'can match or even surpass' larger models rests on the accuracy comparison in Figure 10. The accuracy metric defined in Section 4 checks only the first 3–4 tokens. This is a stringent parsing rule that can classify a semantically correct but verbose output as incorrect. Since the paper does not report error bars or significance tests, the apparent advantage of the 30–124M custom models may be driven by this parsing rule rather than by genuine task competence. This concern is load-bearing because the abstract and conclusion make a comparative claim ('outperform their larger counterparts', 'match or even surpass'). If the large models' accuracies are underestimated by 5–10 percentage points due to format noncompliance, the headline may invert. The proposed test—re-extracting labels from full outputs—is cheap and decisive. It does not require new data collection or training; it only requires the saved generation outputs, which should be released with the code. Why not fully agree with the reader? The representativeness concern affects how broadly the conclusion can be drawn, but it does not challenge the internal validity of the comparison for the tested tasks. A biased metric, if present, would invalidate the comparison itself. Thus the most load-bearing assumption is the fairness of the accuracy measurement, not the representativeness of the datasets. That said, both issues need to be addressed in a revision, which is consistent with the reader's CONDITIONAL verdict.","tokens_in":18807,"tokens_out":7819,"duration_ms":66654,"concrete_test":"Re-run the evaluation in Figure 10 with a lenient extraction rule: count a response as correct if the expected class label appears anywhere in the generated text (after removing line breaks and normalizing whitespace), instead of only within the first 3–4 tokens. Compare accuracy before and after for each model, and record the fraction of outputs where the label appears beyond position 4. If the relative ordering changes—e.g., Llama 3 or Phi 3 accuracy rises enough to match or exceed the custom 124M model—then the 'match or surpass' claim is an artifact of the strict parsing rule. If the ordering is unchanged, the strict metric does not explain the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4, the paper defines accuracy by checking for the expected label within the first 3–4 generated tokens, ignoring line breaks. Although the Alpaca-style prompt instructs models to 'Give your answer only as Tap, Double, or Hold', off-the-shelf models fine-tuned with LoRA on tiny datasets (630 or 350 samples) may not strictly comply: they can produce a correct label later in the output (e.g., 'Based on the sensor data, the gesture is Hold'). Such outputs would be counted as incorrect under the paper's rule even though they are semantically correct. This creates a systematic bias in Figure 10: small custom models, pre-trained on sensor data and fine-tuned on the exact template, are more likely to emit the bare label, whereas larger models (Phi, Llama) may emit extra words, lowering their measured accuracy. Because the central claim is that ~30–124M models can match or surpass much larger models, this measurement artifact would directly undermine the headline result. The paper reports no error bars or statistical tests for these accuracy differences, so the stated superiority may not be significant even before accounting for parsing bias. A concrete check—re-extracting labels from full outputs—can settle whether the comparison is fair.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TinyLLM, a framework for pre-training small GPT-2-style language models (30-124M parameters) on mixtures of general web text and sensor data, fine-tuning them with LoRA on Alpaca-style prompt templates, and deploying them on single-board computers. The evaluation covers three sensing tasks: two in-house datasets (hand-gesture detection with 630 samples and indoor localization with 350 samples) and one external swimming-style dataset (3,730 samples), comparing accuracy, token generation rate, and inference time against fine-tuned Phi-2/3 and Llama-2/3 models. The central claim is that careful data curation allows these small custom models to match or surpass much larger off-the-shelf models on specific embedded sensing tasks while enabling faster local inference.","tokens_in":19076,"tokens_out":6023,"duration_ms":42263,"significance":"If fully supported, the central comparison would be a useful contribution to edge ML: the paper gives a concrete recipe for training and deploying sub-125M models locally, includes a systematic study of the web-to-sensor data mix (Fig. 11), and provides deployment measurements across three SBCs, multiple concurrent instances, and varied background loads (Figs. 13-17). These deployment results are the strongest part of the paper. The accuracy evidence for the headline 'surpass larger models' claim is currently weakened by the first-3-4-token accuracy rule, the absence of error bars and significance tests on the accuracy plots, the post hoc selection of the preferred data split, and the restricted swimming benchmark that uses only 3 of 11 data streams. The paper also ships a framework website and states training costs explicitly, which aids reproducibility. On balance, the central claim is defensible but needs additional evidence and a more careful comparison before it can be accepted at the level stated in the abstract and conclusion.","major_comments":[{"comment":"The accuracy rule defined in §4 - checking for the expected label within the first 3-4 generated tokens, ignoring line breaks - creates a systematic bias against models that produce verbose but correct answers. Fine-tuned off-the-shelf models can emit a correct label after a preamble such as 'Based on the sensor data, the gesture is...', which would be counted as incorrect, while custom models trained on the exact Alpaca template are more likely to emit the bare label first. Because the headline claim rests on Fig. 10, please re-evaluate all models by extracting the label from the full generated output, report both the token-window metric and a full-output metric, or show explicitly that no correct labels occur outside the first 3-4 tokens.","section":"§4, accuracy definition"},{"comment":"The experiment setup states that ten trials are run per configuration and that averages and standard deviations are plotted unless specified otherwise, but Figures 10-12 show no error bars or confidence intervals, and no statistical tests are reported. Several custom-vs-large differences are small (for example, the custom 124M model versus Phi-3 in Fig. 10b), so without variance information or a paired test the superiority claim is not established. Please add per-model error bars or confidence intervals on the accuracy figures and run a paired comparison (for example, a McNemar test on test predictions) for the key custom-versus-large contrasts.","section":"§4, Figures 10-12"},{"comment":"The 'Preferred Split' in Figure 11 is highlighted after inspecting the accuracy curve, and the text itself notes that the 0.9 split on the localization dataset reaches 33% accuracy only because the model consistently returns a single label for all prompts. This makes the claim that a balanced mix is optimal a post hoc selection rather than a tested hypothesis. Please select the split by a pre-specified rule (for example, lowest validation loss) and then evaluate on a held-out test split, or report results across multiple fine-tuning seeds for each split so that the reader can assess variability.","section":"§4.2, Figure 11"},{"comment":"The swimming-style evaluation uses only 3 of the 11 available data streams and achieves a maximum F1-score of 0.78, well below the 0.97 reported by Brunner et al. [5], as the paper acknowledges. This weakens the conclusion that small models match or surpass larger and dedicated systems 'across various applications.' Please either evaluate on the full set of streams with an appropriate context-window representation, or clearly present the swimming result as a reduced-stream ablation and compare against a dedicated baseline trained on the same 3 streams.","section":"§4.3, Table 2 and swimming dataset"},{"comment":"Because LoRA fine-tuning uses only 630 and 350 samples for the two in-house datasets, adherence to the instruction 'Give your answer only as Tap, Double, or Hold' is itself a variable that affects the reported accuracy. The paper does not report how often each model violates this output format, nor how sensitive the results are to the broad LoRA hyperparameter ranges listed in §3.5 (rank 16-256, dropout 0.1-0.3, learning rate 4e-4 to 6e-4). Please add an instruction-following breakdown per model (bare label, label with preamble, gibberish or repeated prompt) and a small hyperparameter sensitivity study for at least the 124M custom model.","section":"§3.4 and §3.5, fine-tuning sensitivity"}],"minor_comments":[{"comment":"The acronym is spelled 'LoRa' in one place in §1 and 'LoRA' elsewhere; please make the spelling consistent throughout.","section":"§1, §3.4"},{"comment":"The text says the framework builds on 'llm.c' but no citation or repository link is given for that codebase; please add one.","section":"§3.3"},{"comment":"The text explains that Phi 3_q2 achieved zero accuracy, but Figure 10 omits this point entirely; please state it in the caption or show it as a 0% point so the reader can see the full comparison.","section":"§4.1, Figure 10"},{"comment":"The artificial 33% accuracy spike at split 0.9 for the localization dataset is discussed only in the text; please mark it in Figure 11 or note it in the caption.","section":"§4.2, Figure 11"},{"comment":"The text notes that the 30M custom model achieves zero accuracy on the gesture dataset; this is an important result and should be visible in Figure 12 or its caption rather than only in the body text.","section":"§4.3, Figure 12"},{"comment":"The abstract contains the typo 'invocating'; it should be 'invoking'.","section":"Abstract"},{"comment":"The experiments in Figures 15-17 use different prompts and different values of n (number of generated tokens); please state the exact prompt and n in each caption so the token-rate comparisons are reproducible.","section":"§4.4-§4.6"}],"recommendation":"major_revision","confidential_remarks":"This is a systems and empirical study rather than a new method, and the deployment measurements are the most solid contribution. The headline claim that small custom models match or surpass much larger models is currently supported only by the accuracy figures that have the measurement and statistical issues described above. I would not recommend rejection, but the abstract and conclusion should be softened or the claims should be backed by the additional evaluations requested in the major comments. The scope may fit an applied ML or embedded systems venue better than a pure cs.LG venue, but the paper can be made acceptable with the requested revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's real contribution is a practical recipe — pre-train a 30–124M GPT-2 on a curated mix of web and sensor tokens, LoRA-fine-tune on Alpaca-style prompts, deploy via llama.cpp — and the claim that this beats or matches billion-parameter models on the three sensor tasks is plausible but not proven. The experiments are honest and the limitations section is candid, but the evidence base is too thin for the headline.\n\nWhat's new: the systematic sweep over model size (30–124M) and sensor/web pre-training mix (0 to 1.0) on three tasks, including two new in-house datasets. That's a legitimate, incremental extension of existing work (GPT-2, Phi/TinyStories, IoT-LM, LoRA, llama.cpp). The deployment measurements on three SBCs are useful, and the multiple-instance/background-load experiments answer a real question for edge use. The paper also points to a framework website and gives enough detail to reproduce the pipeline. That counts for something.\n\nSoft spots, in order. One: the accuracy metric checks only the first 3–4 tokens for the expected label. The stress-test worry about verbose correct outputs is real: a fine-tuned Phi/Llama might say 'Based on the sensor data, the gesture is Hold' and be counted wrong. Because all models are fine-tuned on the same template, the bias may be reduced, but the paper shows no error bars on Figures 10–12 and no full-output parsing to rule it out. That directly taxes the 'surpass' claim. Two: the preferred 0.4–0.6 split in Figure 11 is selected after looking at results, with no held-out confirmation. Three: the swimming benchmark uses 3 of 11 data streams and gets F1 0.78 vs 0.97 for the dedicated model; the paper admits this, but a reader should weigh it. Four: training description says 10B tokens over ~20K steps with batch 64 and seq 1024, which is ~1.3B tokens — an arithmetic inconsistency that needs correction. Five: small datasets (630, 350, 3,730 samples) and no non-LLM baseline make the accuracy comparisons fragile.\n\nThe citation pattern looks fine; no invented entities or circular derivation. The central claim is plausible, but the evidence is conditional.\n\nWho this is for: people building local sensor-interaction models or evaluating small LMs at the edge. It deserves a serious referee — the recipe is concrete and reproducible enough to warrant a revision cycle, not a desk reject. My own recommendation: engage with it, but ask for corrected reporting, error bars, full-output parsing, and at least one classical baseline before relying on the superiority claim.","headline":"A useful framework recipe for small task-specific LMs at the edge, with a plausible central claim that its own small, error-bar-free evaluation doesn't yet fully support.","tokens_in":19623,"tokens_out":2827,"would_cite":false,"duration_ms":18753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that carefully pre-trained and LoRA fine-tuned language models of 30-124M parameters can match or beat billion-parameter models on embedded sensor classification, and can run quickly on single-board computers.","keywords":["small language models","edge inference","sensor data classification","pre-training data curation","LoRA fine-tuning","GPT-2","single-board computers","embedded sensing"],"falsifier":"Run the same fine-tuning recipe on several held-out sensor datasets collected from different users, devices, lighting conditions, and locations than the ones used here (for example, public human-activity benchmarks with more classes and varied sensor placements), and compare 30-124M models against the largest fine-tuned baselines under identical prompt templates. If the small models' label-in-first-tokens accuracy falls well below the larger models' on datasets outside the original three, or if a small model again scores zero on a task with short sensor windows, the claim that small curated models are broadly sufficient for embedded sensing fails.","tokens_in":1742,"feed_emoji":"📡","tokens_out":2182,"duration_ms":50703,"temperature":0.7,"pith_summary":"TinyLLM asks whether the scaling trend in language models can be reversed for embedded sensing: instead of relying on billion-parameter models served over the network, the authors pre-train GPT-2-style models of 30-124M parameters on a curated mix of web text and sensor data, then LoRA fine-tune them on task-specific prompts. They report that these small models match or exceed the accuracy of much larger models (Phi-2/3, Llama-2/3) on hand-gesture, indoor-localisation, and swimming-style classification, while generating tokens far faster on single-board computers. The practical stake is that sensor data can be classified locally on devices with a few gigabytes of RAM, avoiding the latency, connectivity, and privacy problems of remote inference. The paper is an extension claim: it shows a recipe, not a law, proposing that careful data curation, rather than scale, carries accuracy on these tasks.","feed_headline":"Tiny 124M-parameter models match billion-parameter LLMs","feed_subtitle":"Curated sensor data plus LoRA fine-tuning lets 30-124M models run accurately on single-board computers.","key_machinery":"The load-bearing mechanism is a GPT-2-style decoder-only transformer whose size is set by depth $l$, with hidden size $C=64l$ and parameters roughly $N=0.05l^3+3.2l$ million, spanning 30M to 124M parameters. It is pre-trained on a corpus that mixes FineWeb web text with SHL and ExtraSensory sensor data in user-chosen proportions (40:60 by default), then LoRA fine-tuned on Alpaca-style instruction/input/response prompts with sensor readings serialized to integers in a compact range so they fit the 1024-token window. The task reduces to next-token prediction: the correct label must appear within the first few generated tokens, and accuracy is scored exactly that way.","core_discovery":"The paper claims that by controlling the pre-training data and then fine-tuning, models with only tens of millions of parameters can be sufficient for embedded sensing. Concretely, GPT-2-style models of 30M to 124M parameters, pre-trained on a mixture of general web text and curated sensor data, then fine-tuned with LoRA on Alpaca-style prompt/response pairs, match or surpass the classification accuracy of much larger fine-tuned models on gesture, localisation, and swimming-style tasks. The authors also show these small models run locally on single-board computers with higher token rates than the larger models, and that a balanced sensor/web pre-training mix matters: pure web or pure sensor data degrades accuracy. The paper is candid about limits: only GPT-2-style architectures can be pre-trained, the context window is 1024 tokens, and on the one external dataset the top F1 of 0.78 trails the 0.97 of the original swimming-style study, in part because only 3 of 11 data streams fit the context window.","pith_inferences":["If the data-mix finding generalizes, the optimal recipe for other domains may be a majority of in-domain tokens rather than pure domain data; the paper only tests sensor/web splits on two tasks, so testing whether the 40:60 optimum shifts with model size or task would be a natural next step.","The 30M model's zero accuracy on gestures, alongside its highest token rate, suggests a practical Pareto frontier between accuracy and speed; a deployment tool could expose that trade-off instead of prescribing a single model size.","Because the framework converts sensor readings to compact text and uses next-token prediction, the same pipeline could be applied to other serialized time-series modalities such as audio features, IMU streams, or ECG, which the paper does not test.","The 'label in first tokens' scoring rule means the reported accuracy is tied to the chosen prompt template; a user changing the phrasing or adding distractor text could shift results, so the framework's real-world robustness depends on the template staying close to the training distribution."],"forward_implications":["Models of 30-124M parameters can be trained on a single H100 GPU in about nine hours, so sensor applications no longer necessarily require cloud-hosted inference.","On constrained single-board computers, the small custom models still generate several tokens per second even where off-the-shelf models cannot be deployed, and multiple specialised models can run concurrently on a mid-range board.","A balanced mix of sensor data and general web text yields better fine-tuned accuracy than either extreme, suggesting that domain tokens should be a substantial minority of pre-training data.","LoRA fine-tuning requires relatively few examples, e.g. 440 gesture samples, which makes it practical to adapt a custom model to a new sensing task.","Larger models are not uniformly better on these tasks: custom 124M and 101M models match or beat several Phi and Llama variants on gesture and localisation accuracy."],"supporting_citations":[{"why":"Supplies the GPT-2 decoder-only architecture on which the custom 30-124M models are based.","marker":"[41]"},{"why":"Provides the FineWeb corpus used for the general web-text portion of pre-training.","marker":"[39]"},{"why":"Supplies the SHL locomotion dataset used as sensor pre-training data.","marker":"[18]"},{"why":"Supplies the ExtraSensory multi-sensor dataset used as sensor pre-training data.","marker":"[55]"},{"why":"Provides the external swimming-style evaluation dataset and the baseline accuracy the paper compares against.","marker":"[5]"},{"why":"Supplies the LoRA fine-tuning method that the framework uses to adapt small models with few examples.","marker":"[21]"},{"why":"Supplies the Alpaca instruction/input/response prompt template used for fine-tuning and evaluation.","marker":"[51]"},{"why":"Supplies the llama.cpp runtime used to measure token rates and deploy GGUF-converted models on single-board computers.","marker":"[16]"},{"why":"Provides the Llama 3 model used as one of the larger comparison baselines in the accuracy and speed evaluations.","marker":"[11]"}],"fun_headline_variants":["124M edge model beats 1B+ LLM on specific tasks","Curated data lets 30M-124M models rival billion-parameter LLMs","Edge LLMs: Small models match big ones with right training data","TinyLLM: 30-124M models run locally, beat billion-param"],"cache_read_input_tokens":21760,"weakest_assumption_plain":"The entire comparison rests on whether the three evaluation datasets (630 gesture samples, 350 localisation samples, and 3,730 swimming samples, only one from outside the authors' lab) represent real embedded sensing workloads; if those tasks are not representative, the conclusion that small models are broadly sufficient for edge sensing collapses.","fun_headline_variants_meta":{"raw":{"variants":["124M edge model beats 1B+ LLM on specific tasks","Curated data lets 30M-124M models rival billion-parameter LLMs","Edge LLMs: Small models match big ones with right training data","TinyLLM: 30-124M models run locally, beat billion-param"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2650,"prompt_tokens":960,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1605}},"tokens_in":576,"tokens_out":1690,"duration_ms":11201,"temperature":1.0,"reasoning_tokens":1605,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:54:43.548538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning recipe on several held-out sensor datasets collected from different users, devices, lighting conditions, and locations than the ones used here (for example, public human-activity benchmarks with more classes and varied sensor placements), and compare 30-124M models against the largest fine-tuned baselines under identical prompt templates. If the small models' label-in-first-tokens accuracy falls well below the larger models' on datasets outside the original three, or if a small model again scores zero on a task with short sensor windows, the claim that small curated models are broadly sufficient for embedded sensing fails.","supporting_citations":[{"cited_title":"Language Models are Unsupervised Multitask Learners","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 decoder-only architecture on which the custom 30-124M models are based."},{"cited_title":"FineWeb, 04 2024","cited_arxiv_id":null,"evidence_quote":"Provides the FineWeb corpus used for the general web-text portion of pre-training."},{"cited_title":"The university of sussex-huawei locomotion and transportation dataset for multimodal analytics with mobile devices","cited_arxiv_id":null,"evidence_quote":"Supplies the SHL locomotion dataset used as sensor pre-training data."},{"cited_title":"Recognizing de- tailed human context in the wild from smartphones and smartwatches","cited_arxiv_id":null,"evidence_quote":"Supplies the ExtraSensory multi-sensor dataset used as sensor pre-training data."},{"cited_title":"Swimming style recognition and lap counting using a smartwatch and deep learning","cited_arxiv_id":null,"evidence_quote":"Provides the external swimming-style evaluation dataset and the baseline accuracy the paper compares against."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA fine-tuning method that the framework uses to adapt small models with few examples."},{"cited_title":"Hashimoto","cited_arxiv_id":null,"evidence_quote":"Supplies the Alpaca instruction/input/response prompt template used for fine-tuning and evaluation."},{"cited_title":"Llama.cpp.https://github.com/ggerganov/llama.cpp,","cited_arxiv_id":null,"evidence_quote":"Supplies the llama.cpp runtime used to measure token rates and deploy GGUF-converted models on single-board computers."},{"cited_title":"The llama 3 herd of models, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the Llama 3 model used as one of the larger comparison baselines in the accuracy and speed evaluations."}],"review_version":1}