{"id":"f241418b-645f-4c17-bfcf-06903e9cb8e7","arxiv_id":"1909.02803","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On a handwriting recognition task, fitting the model to one user's data via curriculum schedules or similar-sample augmentation slightly improves that user's accuracy but typically weakens general accuracy.","lead":"The paper compares three training schedules for deep learning classifiers (early training on one person's data, sample weighing, and fine-tuning) and two similarity-based data augmentation schemes. It reports small per-user accuracy gains on a handwriting benchmark, usually at the cost of lower accuracy on the general dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shaping results lack a valid control: baseline never sees target user's data, so gains may be due to exposure to that data rather than curriculum schedule.","rationale":"The paper's central empirical claim has two prongs: curriculum shaping and data grouping. The data-grouping experiments use a target-user-inclusive baseline (D′I with random samples vs. similar samples), so those comparisons are internally valid and support the 'personalized augmentation' part, modulo single-dataset scope. The shaping experiments, however, compare methods that train on the target user's data against a baseline trained on a different individual's data. That confound is load-bearing because the most impressive Table I entries (ES-400 and TL-300 beating baseline on both metrics) are exactly the ones claimed to show curriculum order matters. If the same-data uniform baseline also reaches ~0.983 on DI and ~0.98 on DG, the shaping contribution evaporates. I therefore agree with the reader's weakest_assumption and see no reason to change the CONDITIONAL verdict; the requested rerun is a necessary condition for accepting the shaping claim. I am not raising additional objections about statistical testing or dataset diversity because they are secondary to the control-group problem and the paper does report per-user p-values. The absence of code is a reproducibility concern but not a logical flaw in the argument.","tokens_in":10975,"tokens_out":4500,"duration_ms":49008,"concrete_test":"Rerun the shaping rows of Table I with a baseline trained on DG plus the target user's training split (same samples, batch size, and number of gradient steps as the shaping runs) using uniform batching over the union, and evaluate on the target user's held-out test split. Report per-user accuracy over the same 100 users with a paired test. If ES-n-400 and TL-n-300 no longer beat this control on both Acc. on DI and Acc. on DG, the headline claim about curriculum learning is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV states: \"For shaping, the baseline method was trained on the union of DG and data of one individual DI... For the baseline the evaluation was carried out using data from another individual DI′, ie. DI ≠ DI′.\" Thus, for a target user I, the baseline's training set contains another user's handwriting, not I's training split, while every shaping method (EarlyShape, SampleWeigh, TransferL) trains on I's own training split. The reported improvements on \"Acc. on DI\" (e.g., ES-n-400 .983 and TL-n-300 .983 vs. baseline .980) could therefore be caused simply by the model having seen examples of I's writing, not by the curriculum schedule. This directly undermines the strongest claim in the abstract and in the reader's strongest_claim. The footnote \"Otherwise the baseline would be equivalent to our sample weighing method\" does not justify the design: SW-n-2 intentionally oversamples DI (every second batch), whereas a baseline trained on the union DG ∪ DI with uniform batching is a different, well-defined control that matches the shaping methods' training data without imposing a curriculum. Re-running against that control is necessary before any claim about shaping can be accepted. The data-grouping comparison is less affected, because its baseline is built from the target user's D′I, so the central claim may survive for that component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper discusses personalization of deep learning classifiers. It defines objectives (personalized-data accuracy, global-data accuracy, fairness) and proposes two families of methods: three curriculum/shaping schedules (early shaping, sample weighing, transfer learning) and two data-grouping augmentation strategies (individual grouping and single-sample grouping) driven by an autoencoder latent-space similarity. Experiments on the NIST Special Database 19 use a VGG-style CNN and an autoencoder, measuring accuracy on an individual's test data and on a global test set, plus fairness metrics. The central empirical claim is that both curriculum learning and personalized data augmentation improve performance on an individual's data, usually at the cost of global-data performance, with two shaping configurations improving both metrics.","tokens_in":11262,"tokens_out":5642,"duration_ms":61802,"significance":"If the empirical claims hold, the paper would offer a structured comparison of practical personalization techniques and a useful framing of the personalization/global-performance trade-off with fairness metrics. The data-grouping portion uses a reasonable baseline (augmenting the target user's data with random global samples) and reports consistent small gains on individual accuracy. The paper also makes an honest attempt to report variance across users via standard deviations and min/max spreads. However, the shaping evaluations rest on a confounded baseline that never sees the target user's training data, the statistical reporting is too sparse to support the stated significance claims, and all conclusions come from a single dataset and architecture without released code, weakening the generality of the conclusions.","major_comments":[{"comment":"The shaping baseline is not a valid control. The text states that for shaping, the baseline was trained on the union of DG and the data of one individual D_I', while evaluation was carried out on data from another individual D_I' with D_I' != D_I; in contrast, every shaping method (EarlyShape, SampleWeigh, TransferL) is trained on the target user's own training split. The reported improvements on Acc. on DI (e.g., ES-n-400 at .983 and TL-n-300 at .983 versus the baseline at .980) could therefore be caused simply by the model having seen examples of the target user's handwriting rather than by the curriculum schedule. The footnote that otherwise the baseline would be equivalent to the sample weighing method is not convincing: a baseline trained on the union DG ∪ D_I with uniform batching is a well-defined control and is not equivalent to SW-n-2, which deliberately oversamples D_I every second batch. The shaping experiments must be rerun against a control that is trained on the same data as the shaping methods, with only the ordering/weighing changed, before any claim about the benefit of shaping can be accepted.","section":"Section IV (Evaluation, shaping baseline)"},{"comment":"The statistical support for the main claims is insufficient. The paper reports p-value thresholds such as '< 0.1' and '< 0.01' without stating the test used, the number of paired observations, whether corrections for multiple comparisons were applied, or the effect size. Several of the headline comparisons (for example, .983 versus .980 on Acc. on DI for shaping, or .979 versus .975 for data grouping) are small absolute differences and are only claimed significant at the p < 0.1 level, which is above the conventional 0.05 threshold. Please provide full test details, confidence intervals, and a multiple-comparison-corrected analysis, and clarify whether these comparisons are across the 100 users or across models.","section":"Section IV (Results and Table I)"},{"comment":"The methods are repeatedly described as 'shown in Algorithm 1' (Section III, subsections A, B, C, and the data-grouping paragraph), but Algorithm 1 is not present in the manuscript. The pseudo-code is central to specifying the training schedules, sample-weighing rule, and the two grouping procedures, so its absence is a substantial reproducibility gap. Either the algorithm must be included, or the text must fully specify these procedures without relying on the missing reference.","section":"Section III (References to Algorithm 1)"},{"comment":"The paper draws broad conclusions about personalization from experiments on a single dataset (NIST Special Database 19), a single network architecture, and one variation type (hidden factors, not class-count variation). The abstract and conclusions state general claims such as 'shaping can outperform a baseline on both data' and 'data augmentation by choosing similar samples also increases performance of an individual.' These claims need support from at least one additional dataset or task, or the claims need to be explicitly scoped to the conditions tested. In addition, the absence of released code, random seeds, and exact hyperparameter values for the 'more than 1000 trained models' makes independent verification difficult.","section":"Section IV (Evaluation setup and conclusions)"}],"minor_comments":[{"comment":"The abstract contains the typo 'curriculuum learning' and the index terms include 'Artificial,Feature Shaping, Intelligence,' which appears to be an artifact of misformatted keywords; please clean these up.","section":"Abstract and Index Terms"},{"comment":"Section III says the similarity representation is obtained from an auto-encoder 'trained on all data,' while Section IV says the auto-encoder is trained on DG (and on DI only for sample weighing). Please clarify whether any part of the target user's data, including its test split, is used to train the auto-encoder, since this matters for the data-grouping evaluation.","section":"Section III (Data grouping) versus Section IV (Evaluation)"},{"comment":"The notation in Table I (ES-n-400, SW-n-12, TL-n-300, IG-n-5, SG-n-1500, etc.) is not explained before or in the table itself; please add a table note defining n for sample weighing, early shaping, transfer learning, and grouping, and stating the sizes of the augmented datasets for individual grouping.","section":"Table I"},{"comment":"The statement that 'Accuracy correlates with fairness' is made largely on the basis of the summarized rows in Table I; please either report a quantitative correlation (e.g., across the 100 users) or soften the claim, since the table only shows aggregate standard deviations and min/max values for a few configurations.","section":"Section IV (Fairness discussion)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonably concise empirical study, but the shaping results, which support the strongest claimed contribution, are currently undermined by a baseline that does not see the target user's data. Because the fix requires rerunning the experiments with a properly matched control, a major revision is appropriate rather than acceptance. The data-grouping component is much less affected and could form the core of a publishable paper after the statistical reporting and reproducibility gaps are addressed. I would also note that the paper would benefit from positioning against more recent personalization/federated-learning literature, though the current references are adequate for the stated scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the stress-test note is right, and I'd widen it: the control problem affects both halves of the paper, not just shaping. The shaping baseline trains on DG plus one other user's data and tests on the target user; every shaping method trains on the target user's own training split. The data-grouping baseline is similar: it uses random global samples rather than the target user's data. With gains around half a percent, exposure to the target user's data explains the results as plausibly as the curriculum schedule or similarity grouping. The footnote that a different baseline would be equivalent to sample weighing is wrong: training on the union DG ∪ DI with uniform batching is a well-defined control, and it is the one the paper needs.\n\nSecond, the paper does have merit. It lays out objectives and metrics for personalization—individual accuracy, global accuracy, and spread as a fairness proxy—clearly. The systematic comparison of early shaping, sample weighing, and transfer learning is useful, and early shaping (personal data first, then global) is a less-studied variant worth isolating. The autoencoder-based similarity grouping is a reasonable mechanism. Related work is adequate and self-citations are minor.\n\nSoft spots beyond the control: one dataset (NIST digits with flips), no code, one trained model per configuration so there is no seed variance, and reported p-values are mostly above 0.05 with no multiple-comparison correction. The best gains are small (.983 vs .980). So even the data-grouping component, which might survive a re-run, is thin evidence.\n\nWho this is for: practitioners wanting cheap personalization tricks might skim it, but they should not act on the numbers until the baselines are fixed. A student could use the taxonomy and the early-shaping idea as a starting point.\n\nMy recommendation: don't accept as is. It deserves a serious referee only if the authors rerun both experiments against a uniform-batch baseline trained on the same data, report per-seed variance, and ideally add a second dataset. That is a fixable paper, but the headline claim is currently unsupported.","headline":"Useful taxonomy and early-shaping idea, but the headline comparison is confounded because the baseline never trains on the target user's data.","tokens_in":11732,"tokens_out":5110,"would_cite":false,"duration_ms":54482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Personalized training can beat a generic model on both metrics","keywords":["personalization","curriculum learning","transfer learning","data augmentation","autoencoder","deep learning","fairness","handwriting recognition"],"falsifier":"Train the baseline on the global dataset plus the target user's own training data and evaluate on her held-out test set. If that baseline reaches or exceeds the accuracy of EarlyShape-ES-n-400 and TransferL-TL-n-300, then the reported advantage comes from exposure to the user's data, not from the curriculum schedule.","tokens_in":10791,"feed_emoji":"🎯","tokens_out":6776,"duration_ms":63384,"temperature":0.7,"pith_summary":"This paper tries to establish that a deep learning model can be steered toward a particular individual by changing when and how often that individual's data is used during training, or by enlarging her small dataset with similar examples from a shared pool. On a handwritten-digit recognition task with 100 users, the authors show that three curriculum schedules (early shaping, sample weighing, late shaping/transfer learning) and two autoencoder-based data-grouping methods raise accuracy on the individual's own test data compared with a baseline, usually at the cost of lower accuracy on the general dataset. The notable exception is that two configurations, early shaping with 400 initial individual-data iterations and transfer learning with 300 iterations, improve accuracy on both the individual's data and the general dataset at once. The paper also reports fairness as spread in accuracy across individuals and finds that better-performing models tend to have smaller spread. A sympathetic reader would take away that personalization and general performance are in tension, but not strictly so: a short, well-placed dose of individual data can help both.","feed_headline":"Personalized training can beat a generic model on both metrics","feed_subtitle":"With only 400 or 300 extra steps on one user's data, a handwritten-digit model improves on both the user's test set and the global set.","key_machinery":"The carrying objects are the three shaping schedules and the autoencoder-based similarity measure. Shaping is a curriculum over data sources: batches are drawn entirely from either the individual's dataset or the global dataset, and the schedule determines when the individual's data dominates (early shaping), how often it appears (sample weighing), or whether it is used only for final fine-tuning (transfer learning). The similarity machinery is an autoencoder trained on all data; its 16-dimensional encoder output gives a latent representation, and Euclidean distance in that space ranks how close a global sample is to a user's sample (single-sample grouping) or how close one user's class-mean profile is to another user's (individual grouping). This lets a small personal dataset be augmented with look-alike data from the shared pool, which is what makes the augmentation 'personalized.'","core_discovery":"The central claim is that personalization of a deep classifier can be achieved by curriculum shaping and by similarity-based data grouping, and that these are not merely trade-off maneuvers. Shaping changes the exposure schedule: early shaping trains only on the individual's data for a limited number of iterations before switching to the global set; sample weighing interleaves individual batches throughout training (one every n_I-th iteration); transfer learning fine-tunes a globally trained model on the individual's data. Data grouping replaces the individual's small dataset with an enlarged one, where added samples are either the most similar single samples in the global set or entire datasets of similar individuals, with similarity measured by Euclidean distance in a 16-dimensional autoencoder latent space. The paper's key experimental result is that EarlyShape with 400 initial iterations and TransferL with 300 iterations outperform the baseline on both the individual's held-out accuracy (0.983 vs 0.98 and 0.983 vs 0.98) and general accuracy (0.98 vs 0.979 and 0.979 vs 0.979), while sample weighing achieves the best personal accuracy (0.984) but significantly worse general accuracy (0.968).","pith_inferences":["The paper's own evaluation section states that the baseline is tested on a different individual's data; that makes it an open question whether the shaping gains come from the schedule or from exposure to one real user's data, and a direct control would train the baseline on the target user's own training data.","The same autoencoder similarity could support privacy-preserving personalization: instead of sharing raw data, only latent representations of similar samples would need to be exchanged, though the paper does not explore this.","The class-count variation scenario was explicitly deferred, so the methods' behavior when individuals differ in how many samples they have per class remains an open testable question.","A practical extension would be to treat the number of initial or final individual-data iterations as a tunable knob and study how personal and general accuracy trade off as that number grows, which the paper only samples at a few points."],"forward_implications":["Users can get a model tuned to their own handwriting with a short curriculum phase, without retraining from scratch for everyone.","If the two winning configurations replicate beyond this dataset, early and late shaping are both viable personalization strategies that do not force a personalization-versus-generalization sacrifice.","The more aggressively a model is shaped toward one individual (sample weighing), the better it does on that individual and the worse on the general set, so deployment needs a tunable exposure parameter.","Augmenting an individual's data with only the closest similar samples improves personal accuracy; adding too many or more diverse samples shifts performance back toward the general baseline.","Reporting minimum, maximum and standard deviation of accuracy across individuals gives a practical fairness check: models with higher average accuracy also showed less spread."],"supporting_citations":[{"why":"Provides the VGG-style convolutional architecture used for all classification experiments.","marker":"[18]"},{"why":"Introduces curriculum learning, the ordering principle behind the shaping schedules.","marker":"[31]"},{"why":"Defines transfer learning and fine-tuning, which the late-shaping variant instantiates.","marker":"[8]"},{"why":"Supplies the psychological notion of shaping that motivates early and late exposure to individual data.","marker":"[7]"},{"why":"Documents catastrophic forgetting, the failure mode the paper must avoid when switching datasets.","marker":"[16]"},{"why":"Shows autoencoder-based data augmentation, the family of techniques the data-grouping methods extend.","marker":"[40]"},{"why":"Uses an autoencoder for curriculum learning, motivating the latent-space similarity for grouping.","marker":"[41]"}],"fun_headline_variants":["Curriculum and data grouping lift both user and global accuracy","Early shaping and transfer learning beat generic on both metrics","Two personalization tricks improve personal and global accuracy","Shaping and grouping: personalization without a generality hit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that a baseline trained on the global dataset plus one individual's data and tested on a different individual is a fair control, so the gains of the shaping methods are attributed to the training schedule rather than to having seen a real user's data during training.","fun_headline_variants_meta":{"raw":{"variants":["Curriculum and data grouping lift both user and global accuracy","Early shaping and transfer learning beat generic on both metrics","Two personalization tricks improve personal and global accuracy","Shaping and grouping: personalization without a generality hit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3246,"prompt_tokens":892,"completion_tokens":2354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2291}},"tokens_in":508,"tokens_out":2354,"duration_ms":19764,"temperature":1.0,"reasoning_tokens":2291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:37:50.241256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the baseline on the global dataset plus the target user's own training data and evaluate on her held-out test set. If that baseline reaches or exceeds the accuracy of EarlyShape-ES-n-400 and TransferL-TL-n-300, then the reported advantage comes from exposure to the user's data, not from the curriculum schedule.","supporting_citations":[{"cited_title":"Curriculum learning,","cited_arxiv_id":null,"evidence_quote":"Introduces curriculum learning, the ordering principle behind the shaping schedules."},{"cited_title":"Deep learning of representations for unsupervised and transfer learning,","cited_arxiv_id":null,"evidence_quote":"Defines transfer learning and fine-tuning, which the late-shaping variant instantiates."},{"cited_title":"A day of great illumination: B.f. skinner’s discovery of shaping,","cited_arxiv_id":null,"evidence_quote":"Supplies the psychological notion of shaping that motivates early and late exposure to individual data."},{"cited_title":"Overcoming catastrophic forgetting in neural networks,","cited_arxiv_id":null,"evidence_quote":"Documents catastrophic forgetting, the failure mode the paper must avoid when switching datasets."},{"cited_title":"Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation,","cited_arxiv_id":null,"evidence_quote":"Shows autoencoder-based data augmentation, the family of techniques the data-grouping methods extend."},{"cited_title":"Autoencoder-based semi- supervised curriculum learning for out-of-domain speaker verification,","cited_arxiv_id":null,"evidence_quote":"Uses an autoencoder for curriculum learning, motivating the latent-space similarity for grouping."}],"review_version":1}