{"id":"fa890e6f-6bbe-4ae1-8144-814726cb8631","arxiv_id":"2412.08048","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A small benchmark on a new Indian student face dataset reports higher accuracy for deep models, but methodological gaps make the numbers unverifiable.","lead":"This paper benchmarks classic and deep face recognition models against a new 40-person dataset of Indian college students and compares them with a subset of LFW. The reported results favor CNN and hybrid models, but lack error bars, code, and a clearly identity-disjoint test split.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracies are not trustworthy because Section 4.4 does not state that the 80:20 split is identity-disjoint, allowing the same subjects in both training and test sets and turning recognition into memorization.","rationale":"The reader's weakest assumption targets exactly the missing identity-disjoint split in Section 4.4, and I agree this is the load-bearing issue. Without a group-level split, the accuracy numbers in Section 5 cannot be interpreted as face-recognition generalization; they could reflect image-level memorization, especially because the dataset is tiny (40 subjects, 10 photos each plus augmentation) and the Hybrid model reports 100% accuracy on both datasets. I also note Section 6's 'hypothetical results' as an explicit concession that the empirical claims are not firmly established, but the decisive fix is to disclose and enforce identity-disjoint splits. Additional reporting—error bars, code, ethics statement—would help a revision, but none of it can rescue the central comparison if the split is not identity-disjoint. Thus the reader's REJECT verdict stands without adjustment.","tokens_in":5592,"tokens_out":3480,"duration_ms":37626,"concrete_test":"Re-run the evaluation with an identity-disjoint split: use a GroupShuffleSplit on subject IDs (or equivalent) so all images and augmentations of each subject go exclusively to either training or test, then recompute CNN, Hybrid, and LDA+kNN accuracy on both datasets with the same preprocessing and hyperparameters. The decisive comparison is between this group-level split and the original split: if accuracies drop materially (e.g., Hybrid falls from 100% or LDA+kNN's JFAD gain over LFW shrinks by more than a few points), the headline numbers reflect image-level leakage rather than generalized face recognition. If possible, also report mean +/- std over 10 random seeds for both split schemes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (Section 5.3: CNN 97.64%->98.404%, Hybrid 100%, LDA+kNN 75.66%->87.76%) depends on training and test splits being identity-disjoint: no subject, or augmented image derived from that subject, may appear in both folds. Section 4.4 says only that 'The datasets were then split into training and testing sets with a 80:20 ratio' after augmentation. Since each subject has 10 images and augmentation creates many more, a random image-level split will place different photos of the same subject in both folds. A face recognizer can then solve the task by memorizing identities rather than by generalizing to unseen people, which is especially plausible for the high-capacity CNN and the modified GoogLeNet Hybrid that both report perfect or near-perfect accuracy. The conclusion that dataset selection changes model rankings is therefore unsupported as stated. Section 6's phrase 'hypothetical results' is an in-text signal that the numbers were not presented as a validated empirical finding, compounding the methodological ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates several face recognition approaches—PCA+kNN, LDA+kNN, LPP, Gabor+kNN, a CNN, and a hybrid Haar/edge+GoogLeNet model—on two datasets: a 50-subject subset of LFW and a newly collected 40-subject Indian student dataset (JFAD). It reports that the CNN and hybrid models achieve near-perfect accuracy on both datasets, and that most models perform better on JFAD than on the LFW subset, with LDA+kNN rising from 75.66% to 87.76%. The authors conclude that dataset selection significantly affects face recognition performance and that the JFAD is a useful benchmark for Indian demographics.","tokens_in":5810,"tokens_out":2753,"duration_ms":28933,"significance":"If the empirical claims were properly supported, the paper would offer a useful contribution: a new Indian-focused face dataset and a comparison of classical versus deep models on it. The motivation to study demographic diversity in face recognition is timely and important. However, the evaluation as described does not establish the central quantitative results. The train/test split is not stated to be identity-disjoint, the results are reported as single runs without error bars, and Section 6 refers to the reported numbers as 'hypothetical results.' These issues affect every comparative claim in Section 5 and prevent the paper from making a trustworthy empirical contribution in its current form.","major_comments":[{"comment":"The experimental procedure does not state that the 80:20 split is identity-disjoint. Section 4.4 says only 'The datasets were then split into training and testing sets with a 80:20 ratio' after augmentation. Since each subject has 10 original images and augmentation multiplies the data, a random image-level split will likely place images of the same subject in both training and testing. In that case, the high accuracies in Section 5.3 (e.g., CNN 98.404%, Hybrid 100%, LDA+kNN 87.76%) can reflect memorization of identities rather than generalization to unseen people. The paper must specify whether the split was by identity and, if not, re-run the experiments with an identity-disjoint split.","section":"Section 4.4 and Section 5.3"},{"comment":"The discussion states that the CNN 'emerged as the most promising in our hypothetical results.' The word 'hypothetical' directly contradicts the empirical framing of Section 5, where the numbers are presented as measured accuracy. The paper needs to clarify whether the results are actual experimental outcomes or illustrative/hypothetical; if they are hypothetical, the central claims of the paper are not supported by data.","section":"Section 6"},{"comment":"All accuracy, precision, recall, and F1 values are reported as single-run point estimates with no variance, confidence intervals, or number of repeated trials. For small datasets (40 and 50 subjects) and high-capacity models, single-run accuracy is not a reliable basis for the comparative ranking claimed in Sections 5.2 and 5.3. The paper should report multiple runs, cross-validation, or at least error bars, and should make the raw numeric results available as a text table rather than only as the image in Figure 7.","section":"Section 5 and Figure 7"},{"comment":"The preprocessing descriptions are incomplete and somewhat inconsistent. Section 3.2 states that JFAD images are resized to 250x250 for comparison, while Section 3.1 says the LFW subset images are 256x256. Section 4.4 reports that after preprocessing JFAD has 2816 images and LFW has 3508 images, but the augmentation parameters (number of augmented images per original, types and ranges of transformations) are not specified. Without these details, the experimental setup cannot be reproduced, which is especially important because the JFAD is claimed to be a new benchmark resource.","section":"Section 3.2 and Section 4.4"}],"minor_comments":[{"comment":"There is a typo in 'correspoinding vector' and inconsistent spelling between 'Minimisation' and the American spellings used elsewhere; these should be corrected.","section":"Section 4.1.1"},{"comment":"Several references appear unrelated to face recognition or dataset diversity, including [1] on ambient assisted living, [8] on COVID/vaccine discourse, and [10] on video action recognition. The paper should cite relevant fairness and face-recognition-benchmark literature instead.","section":"References"},{"comment":"The JFAD dataset is released only as a Google Drive link with no license, no consent or IRB statement, and no demographic metadata beyond age range. For a dataset intended to serve as a benchmark, these details are important and should be added.","section":"Section 3.2"},{"comment":"The use of RetinaFace to remove images where no face is detected is described without quantifying how many images were removed or whether removal rates differed across demographics; this should be reported to assess potential bias in the preprocessing step.","section":"Section 4.4"}],"recommendation":"reject","confidential_remarks":"The paper addresses a relevant topic and the JFAD collection is a potentially useful resource, but the central empirical claims are not currently supportable: the train/test split is not identity-disjoint, results are single runs without uncertainty, and Section 6 itself labels the results as 'hypothetical.' Because these issues are load-bearing for the paper's main conclusions, I cannot recommend acceptance or minor revision. A future resubmission with a proper identity-disjoint evaluation, error bars, and a clarified stance on the empirical status of the numbers could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, its only real contribution is JFAD, a small new dataset of 400 images from 40 IIT Jodhpur students intended as an Indian-demographics testbed. Second, the accuracy numbers in the results section cannot be taken at face value as submitted: the 80:20 split in Section 4.4 is not described as identity-disjoint, and Section 6 refers to \"hypothetical results,\" which effectively concedes the numbers aren't a validated empirical finding.\n\nWhat's genuinely new: the dataset itself, and the comparison of standard models (Eigenfaces, LDA, LPP, Gabor, CNN, hybrid GoogLeNet) on both LFW and JFAD. If the dataset is actually released and documented, it could have modest value for the fairness-in-face-recognition literature, which does need more Indian data. The paper also presents the basic methodology clearly enough for a reader to see what was attempted.\n\nThe soft spots are serious. The train/test split is the load-bearing one. With ten images per subject and augmentation multiplying them, a random split places the same identities in both training and test sets; a high-capacity CNN or the hybrid model can then memorize subjects and report near-perfect accuracy without generalizing. The absence of any statement that splits are disjoint by identity, plus the \"hypothetical results\" phrase, makes the central comparative claim (LDA+kNN jumps from 75.66% to 87.76%, etc.) unsupported as stated. There are also no error bars or multiple runs, no code or hyperparameter details, and no consent or ethics statement for identifiable student photos. These aren't minor quibbles; they affect whether the empirical claims mean anything.\n\nThe citation list is also weak — references [1], [8], and [10] appear unrelated to face recognition — though that's fixable and not central.\n\nIn short: this is a draft that could become a small benchmark note if the authors release the data, specify identity-disjoint splits, run repeated trials, report variance, and add an ethics statement. As submitted, the results are not trustworthy enough to build on.\n\nMy recommendation for peer review: desk reject, but with a clear resubmission path. It doesn't deserve referee time in its current form.","headline":"A small new Indian face dataset is the only real contribution; the benchmark numbers are not reliable as reported because the split may leak identities and the authors call the results 'hypothetical'.","tokens_in":6331,"tokens_out":3040,"would_cite":false,"duration_ms":30311,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new Indian student face dataset shifts how recognition models rank, lifting LDA+kNN from 75.66% to 87.76%.","keywords":["facial recognition","Indian demographics","JFAD","LFW","CNN","hybrid model","dataset bias"],"falsifier":"Re-run the same models with a subject-disjoint 80:20 split (all ten images of each subject on one side) and check whether CNN accuracy on JFAD stays near 98.4% and LDA+kNN stays near 87.76%; if accuracy drops substantially, the original numbers were inflated by identity leakage.","tokens_in":5375,"feed_emoji":"🇮🇳","tokens_out":15437,"duration_ms":124590,"temperature":0.7,"pith_summary":"The paper introduces a small custom dataset of Indian student faces, called JFAD, and compares a range of face-recognition models on it against the standard LFW dataset. The authors report that the CNN and hybrid models reach near-perfect accuracy on both datasets, and that most models, especially LDA+kNN, score higher on JFAD than on LFW (LDA+kNN rises from 75.66% to 87.76%). They conclude that dataset selection materially changes recognition performance and that building inclusive, population-specific datasets is important for fair deployment. The work also offers the JFAD collection as a resource for future evaluation.","feed_headline":"New Indian face dataset lifts LDA+kNN from 75.66% to 87.76%","feed_subtitle":"Most models also score higher on the Indian set than on LFW, highlighting the dataset's role in model rankings.","key_machinery":"The load-bearing objects are the two datasets and the comparison protocol. JFAD, the new custom set, contains 400 images of 40 Indian students (ten per subject) captured under varied real-world conditions; the LFW subset mirrors that structure with 50 subjects and ten images each. After face detection, augmentation (rotation, shear, brightness/contrast/sharpness), and resizing, the images are split 80:20 into training and test sets, and each model is evaluated on both datasets. The model suite spans subspace projections (PCA, LDA, LPP), Gabor-filter features, a convolutional network, and a hybrid that combines Haar and edge features with a convolutional classifier. Running the same models on the two datasets is the mechanism that yields the paper's conclusions about dataset selection.","core_discovery":"On the paper's own terms, the central discovery is that the choice of evaluation dataset changes both the absolute accuracy and the ranking of face-recognition models. Using a newly collected set of 400 Indian student portraits (JFAD, expanded to 2,816 images after augmentation) alongside an LFW subset (3,508 images), the authors find that the CNN and a hybrid model that combines Haar and edge features with a convolutional network achieve near-perfect accuracy on both. The linear-discriminant model (LDA+kNN) improves markedly on JFAD, from 75.66% to 87.76%, and the hybrid model attains perfect accuracy on both datasets. The authors interpret these results as evidence that dataset composition—ethnic diversity, capture conditions, and image resolution—strongly affects recognition performance, and that population-specific benchmarks are needed to tune and validate systems for deployments in India.","pith_inferences":["As an editorial extension, because the paper does not state that its 80:20 split keeps all images of a subject on one side, the near-perfect accuracy figures may partly reflect within-person similarity rather than generalization to unseen identities.","With 40 subjects in JFAD and 50 in LFW, the reported accuracy differences carry large sampling uncertainty, so a broader Indian cohort would be needed to confirm the ranking shifts.","The dataset is drawn from a narrow academic age band (18–23), so treating JFAD as representing India's full ethnic and demographic diversity would overstate the paper's evidence.","A direct test would be to re-evaluate all models on a subject-disjoint split and on a more varied multi-region Indian dataset to see whether the LDA+kNN jump and near-perfect CNN/hybrid scores persist."],"forward_implications":["If the results hold, model rankings on LFW cannot be assumed to transfer to Indian populations; the same model can appear weak or strong depending on the benchmark.","The LDA+kNN jump from 75.66% to 87.76% suggests that classical linear methods can become competitive when the training and test faces come from a homogeneous, population-specific set.","The CNN and hybrid models' near-perfect accuracy on both datasets implies that current deep architectures handle this Indian student sample at least as well as the LFW sample.","The authors' conclusion that dataset selection is critical supports building and sharing more region-specific benchmarks and tuning systems per deployment context."],"supporting_citations":[{"why":"supplies the LDA/Fisherfaces subspace method used as a classical baseline.","marker":"[2]"},{"why":"supplies the deep face detector used to crop and clean images in preprocessing.","marker":"[3]"},{"why":"motivates the need for Indian-specific facial recognition datasets that JFAD addresses.","marker":"[4]"},{"why":"provides the convolutional network foundation for the deep learning approach.","marker":"[5]"},{"why":"supplies the Gabor filter feature extraction method used for texture-based recognition.","marker":"[6]"},{"why":"supplies the convolutional backbone used in the hybrid model.","marker":"[9]"}],"fun_headline_variants":["Indian face dataset reshuffles model rankings, boosts LDA+kNN by 12%","JFAD dataset: Indian faces expose benchmark bias in face recognition","LDA+kNN jumps 12% on new Indian face dataset, hybrids hit 100%","New Indian face benchmark reveals dataset-dependent accuracy and rank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume the 80:20 train/test split keeps every person's images entirely in either training or testing, but the paper never states that the split is by identity.","fun_headline_variants_meta":{"raw":{"variants":["Indian face dataset reshuffles model rankings, boosts LDA+kNN by 12%","JFAD dataset: Indian faces expose benchmark bias in face recognition","LDA+kNN jumps 12% on new Indian face dataset, hybrids hit 100%","New Indian face benchmark reveals dataset-dependent accuracy and rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1580,"prompt_tokens":911,"completion_tokens":669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":587}},"tokens_in":527,"tokens_out":669,"duration_ms":6752,"temperature":1.0,"reasoning_tokens":587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:16:33.805390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same models with a subject-disjoint 80:20 split (all ten images of each subject on one side) and check whether CNN accuracy on JFAD stays near 98.4% and LDA+kNN stays near 87.76%; if accuracy drops substantially, the original numbers were inflated by identity leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the LDA/Fisherfaces subspace method used as a classical baseline."},{"cited_title":"Retinaface: Single-stage dense face localisation in the wild","cited_arxiv_id":null,"evidence_quote":"supplies the deep face detector used to crop and clean images in preprocessing."},{"cited_title":"Kumar and P","cited_arxiv_id":null,"evidence_quote":"motivates the need for Indian-specific facial recognition datasets that JFAD addresses."},{"cited_title":"LeCun, L","cited_arxiv_id":null,"evidence_quote":"provides the convolutional network foundation for the deep learning approach."},{"cited_title":"Liu and H","cited_arxiv_id":null,"evidence_quote":"supplies the Gabor filter feature extraction method used for texture-based recognition."},{"cited_title":"Going deeper with convolutions","cited_arxiv_id":null,"evidence_quote":"supplies the convolutional backbone used in the hybrid model."}],"review_version":1}