{"id":"6f291c8b-9663-4a5c-b928-eb731a5062f9","arxiv_id":"2507.15862","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"CAPS fuses LLM essay scores, semantic embeddings, and regression weights into one admissions score, but validates it only on synthetic applicants.","lead":"This paper introduces CAPS, a three-module machine learning score that combines grades, essays, and activities into a single admissions number. The headline accuracy comes only from a synthetic dataset, so it does not show CAPS works on real college applications.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EQI R²=0.80 is likely an artifact: the EQI target is never defined independently of the GPT-4o rubric features used as predictors, so the headline metric may only reflect the model learning its own label-generating function.","rationale":"The paper's headline quantitative achievement is the EQI prediction R²=0.80, presented as evidence that CAPS reproduces human-like essay evaluation. The load-bearing assumption is that EQI is a real, externally meaningful quantity. The manuscript never defines EQI; the only numeric essay-quality signals described are GPT-4o rubric scores that are also the model's most important features. If the target is generated from these same scores, the R² is automatically high and the result is uninformative about human alignment. The reader's weakest assumption correctly flags this; my stress-test narrows it to the EQI module, where the circularity is most direct and most damaging. The synthetic-dataset concern alone would support only a proof-of-concept caveat; the EQI target circularity undermines the central regression result entirely. Because the authors release no code or data, the ambiguity cannot be resolved from the paper. A concrete test—retraining on MiniLM embeddings alone after obtaining the target formula—would settle it. Secondary issues (undefined class names, inconsistent macro-F1) strengthen the impression of insufficient validation, but the EQI circularity is sufficient for the verdict. No change to the reader's REJECT is needed; hence the verdict remains UNCHANGED.","tokens_in":10381,"tokens_out":4801,"duration_ms":49051,"concrete_test":"Request the exact EQI target formula and data-generation script from the authors. If the target is any deterministic function of the three GPT-4o rubric scores, retrain the XGBoost model on the 384 MiniLM embeddings alone (dropping the three rubric-score features). If R² falls from 0.80 to near zero, the reported metric is a leakage artifact; if R² remains high, the circularity concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the EQI prediction R²=0.80, presented as evidence that CAPS reproduces human-like essay evaluation. Yet the EQI target is never defined. In §III-C (EQI Module, Step 3), the feature vector is the concatenation of three GPT-4o rubric scores (Content, Language, Structure) with 384 MiniLM embeddings, and the regressor is 'trained to predict continuous EQI scores in the range [0,1]'. §IV repeats this setup and describes the features in the same way, again with no independent target definition. The only essay-quality evidence described is the same GPT-4o output, plus a vague statement that 'a subset of scores was manually verified' with no labels, no counts, and no inter-rater reliability. If EQI is computed from those rubric scores—e.g., a normalized average—then the regressor has the generating features as inputs, and R²=0.7999 measures how well XGBoost recovers that deterministic mapping, not alignment with human judgment. The paper's Limitations section honestly concedes the synthetic dataset cannot capture real-world complexity, but it does not acknowledge this self-referential target issue. Because the claim 'aligned with human judgment' rests on the EQI regression, a circular target would invalidate the headline result even under the authors' own synthetic-data caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAPS, a modular framework that combines academic, essay, and extracurricular components into a single interpretable admissions score. SAS is computed from normalized academic features with a blend of PCA-derived and expert weights; EQI is obtained by feeding GPT-4o rubric scores concatenated with MiniLM embeddings into an XGBoost regressor; EIS fuses GPT-4o activity scores with expert tiers and a coherence adjustment. The final CAPS is a convex combination of logistic-regression weights, XGBoost importance weights, and expert priors, optionally adjusted by a diversity bonus. Experiments on a synthetic dataset report an EQI prediction R² of 0.80, classification accuracy over 75%, macro F1 of 0.69, and weighted F1 of 0.74, with SHAP used for interpretability.","tokens_in":10730,"tokens_out":4651,"duration_ms":49328,"significance":"The problem addressed—transparency and interpretability in holistic college admissions—is important and timely, and the modular architecture with SHAP-based explanations is a plausible blueprint for applicant-facing feedback systems. The paper honestly concedes in Section V that the synthetic dataset cannot capture real-world complexity, and it makes a useful engineering contribution by combining LLM rubric scoring, semantic embeddings, and structured feature fusion. However, the quantitative claims are not established because the EQI target is circularly defined and the admission-tier labels are synthetic functions of the same features used for prediction. If the framework were re-evaluated on real admissions data with independently defined targets, the approach could be valuable; as presented, the reported metrics do not provide evidence of alignment with human holistic review.","major_comments":[{"comment":"The EQI regression target is never defined independently of the predictor features. In §III-C Step 3 and §IV, the XGBoost regressor is trained on a feature vector formed by concatenating three GPT-4o rubric scores (Content, Language, Structure) with MiniLM embeddings, and it is asked to predict a continuous EQI score in [0,1], yet the paper nowhere states how the EQI target values are computed. If EQI is simply a normalized combination of the same rubric scores, then the R²=0.80 in Eq. (18) measures how well the model recovers its own label-generating function, not agreement with human judgment. The statement that 'a subset of scores was manually verified' provides no counts, no inter-rater reliability, and no evidence that the target is human-aligned.","section":"§III-C, §IV"},{"comment":"The admission-tier labels used for classification evaluation are synthetic and are generated from the same module scores that serve as features. Each applicant is represented by SAS, EQI, and EIS, and the tiers are drawn from these scores; logistic regression is then trained on those three scores to predict the tiers, and the fused CAPS weights in Eq. (14) are also fit to the same synthetic outcomes. Under this design, high accuracy may only confirm that the label-generating mechanism is recoverable from its inputs, which is not evidence that CAPS predicts real holistic admissions decisions.","section":"§IV"},{"comment":"The reported macro F1 of 0.69 is inconsistent with the per-tier results in the same subsection. If the three listed classes are 'Tier 0–2' with F1=1.00, Tier 3 with F1=0.44, and Tier 4 with F1=0.00, the macro average is (1.00+0.44+0.00)/3=0.48, not 0.69. If the macro F1 is computed over five separate tiers, the table must report all five per-class rows. Please provide the full confusion matrix and the exact averaging procedure.","section":"§IV-A"},{"comment":"The XGBoost classifier evaluation reports only training-set performance: 'perfect accuracy on the training set (100%)' with macro and weighted F1 of 1.00. Without held-out test metrics, this result is uninformative about generalization and appears to be an overfitting artifact. The robustness claim based on this result is not supported.","section":"§IV-A(b)"},{"comment":"The 'Human Validation' step is asserted but not documented: no number of essays reviewed, no agreement statistics, and no scoring protocol are given. Since the paper's central claim of human alignment rests on this validation, the absence of any quantitative human-judgment comparison is a load-bearing gap.","section":"§IV"}],"minor_comments":[{"comment":"The diagram title contains the typo 'Comphrehensive'; it should read 'Comprehensive'.","section":"Fig. 2"},{"comment":"The multinomial logistic regression results paragraph is duplicated verbatim; one copy should be removed.","section":"§IV-A"},{"comment":"There are typos in the prompts: 'propmt' in the EIS step and 't he' in the alignment prompt; please proofread the prompt texts.","section":"§III-D, §III-C"},{"comment":"References [2] and [7] have nearly identical titles; please verify the citation metadata and confirm that the two entries are distinct works.","section":"References"}],"recommendation":"reject","confidential_remarks":"I do not see a path to revision within the present scope: the evaluation would need to be redone on real data with independently defined labels, which is effectively a new study rather than a local correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a coherent proof-of-concept for decomposing holistic admissions into academic, essay, and extracurricular scores, but the headline R²=0.80 for essay quality is likely an artifact of a circular target. The EQI score that the XGBoost regressor is trained to predict is never defined independently of the GPT-4o rubric scores used as input features. If EQI is just a normalized average of those rubric scores, then R² measures how well the model recovers its own label-generating function, not alignment with human judgment. The stress-test note has this right.\n\nWhat the paper does well: it assembles existing pieces (MiniLM embeddings, GPT-4o scoring, XGBoost, SHAP, logistic regression) into a named, modular framework, and it is transparent about the synthetic-data limitation. The limitations section is honest that performance should be read as proof-of-concept. The writing is clear and the component decomposition is a sensible way to frame the problem.\n\nThe soft spots are load-bearing. Beyond the EQI circularity, the classification labels are generated from the same module scores used as features, so the 75% accuracy and F1 numbers don't tell us anything about real admissions decisions. No code or data is released, which makes the reported numbers hard to check. Minor points: there's a duplicated paragraph in Section IV, and the reader's claim of a macro F1 inconsistency doesn't hold up—with five tiers where the first three each have F1=1.00, a macro F1 of 0.69 checks out.\n\nThis is a paper for researchers working on admissions prediction and explainable AI in education. It's also a good teaching example of why synthetic labels need independent targets. I would not cite it in my own work until it's evaluated on real data, but I'd send it to a serious referee: the problem is genuine, the framework is coherent, and a revision with real institutional data and a properly defined EQI target could turn this into a useful proof-of-concept. As is, the central claim is unsupported.","headline":"A coherent proof-of-concept whose headline R² is likely circular; send to review with strong revision demands.","tokens_in":11232,"tokens_out":3135,"would_cite":false,"duration_ms":35226,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes CAPS, a modular score that breaks holistic admissions into academic, essay, and extracurricular components and claims these predict admission tiers with over 75% accuracy.","keywords":["college admissions prediction","holistic review","interpretable machine learning","essay quality assessment","transformer embeddings","XGBoost regression","SHAP explanations","multi-modal score fusion"],"falsifier":"Give the pipeline a corpus of real application essays with independent human admissions-officer ratings, build the same 387-dimensional features, and test whether the XGBoost regressor still reaches an R² near 0.80 for EQI; then apply the full CAPS score to a university's historical admit/deny records and compare predicted tiers with actual decisions. A large drop in R² or near-chance tier accuracy would show the synthetic results were artifacts of the generation process.","tokens_in":10166,"feed_emoji":"🎓","tokens_out":5232,"duration_ms":56532,"temperature":0.7,"pith_summary":"College admissions decisions described as holistic are usually made behind closed doors, leaving applicants with little sense of where they stand. This paper argues that the evaluation can be decomposed into three measurable parts—academics, essay quality, and extracurricular impact—and then recombined into a single transparent score. The proposed Comprehensive Applicant Profile Score (CAPS) predicts admission tiers from those parts with over 75% accuracy and predicts essay quality with an R² of 0.80 on a synthetic dataset. If the approach transfers to real applicant pools, applicants could receive concrete, component-level feedback and institutions could audit their own criteria instead of relying on opaque heuristics. The paper is careful to call the results a proof of concept because the dataset is synthetic.","feed_headline":"One explainable score predicts admissions tiers with 75% accuracy","feed_subtitle":"CAPS splits each applicant into academics, essays, and extracurriculars, then ranks them transparently.","key_machinery":"The load-bearing machinery is the three-module decomposition plus a convex score fusion. SAS normalizes academic inputs with z-scores, derives weights from PCA and expert-defined priors, and maps them to a 0–100 scale. EQI encodes essays with all-MiniLM-L6-v2 embeddings and GPT-4o rubric scores, trains an XGBoost regressor on the concatenated 387-dimensional vector, then applies a sigmoid penalty for prompt misalignment. EIS blends GPT-4o impact scores with a five-tier activity ladder and a thematic coherence adjustment. The fusion module combines logistic-regression coefficients, XGBoost importances, and expert priors with tunable weights (α=0.3, β=0.3, γ=0.4), so the final score is an explicit weighted sum rather than a black box. SHAP explains each EQI prediction by feature contribution.","core_discovery":"The central claim is that holistic review is quantifiable: each applicant can be represented by a Standardized Academic Score (SAS), an Essay Quality Index (EQI), and an Extracurricular Impact Score (EIS), and a weighted fusion of these three explains most of the variation in admission tiers. The EQI module is the technical core: GPT-4o scores each essay on content, language, and structure, MiniLM embeddings add 384 semantic dimensions, and an XGBoost regressor combines the 387 features to predict a continuous quality score, reaching an R² of 0.80 on held-out essays. A sigmoid penalty adjusts for prompt alignment, and SHAP attributions show that the three GPT rubric scores dominate the prediction. The final CAPS score is a normalized convex combination of module weights learned from logistic regression, XGBoost feature importance, and expert priors, with an optional diversity bonus capped at 12 points. The paper interprets these results as evidence that an interpretable, modular score can align with human judgment while giving applicants actionable feedback.","pith_inferences":["Beyond the paper: the real test of CAPS is transfer to non-synthetic data with independently decided admission outcomes; we expect the headline metrics to drop unless the real decisions are as linearly separable as the simulated tiers.","Beyond the paper: the EQI module could serve as a feedback instrument even if it never decides admissions, since SHAP-based rationales tell students which essay dimensions to improve.","Beyond the paper: because the EQI target and the strongest predictor features both come from the same GPT-4o rubric, the 0.80 R² partly measures the LLM's self-consistency; an independent human-rated essay corpus would separate self-consistency from true essay quality.","Beyond the paper: the fusion weights (0.3, 0.3, 0.4) are tuned on synthetic data; institutions adopting CAPS would need a principled way to set them from stated policy rather than from one simulated pool."],"forward_implications":["If the central claim holds, a single dashboard could show an applicant their academic, essay, and extracurricular scores separately, turning vague admissions feedback into targeted advice.","Institutions could inspect the fused weights and SHAP contributions to see which factors drive their decisions, making criteria auditable.","The modular design means a university could retrain or re-weight modules on its own historical data instead of adopting a fixed national formula.","The reported R² of 0.80 for EQI would justify using embedding-plus-rubric features as a fast proxy for essay quality before human review.","The complete failure to classify the rare Tier 4 in the synthetic data shows that rare admission outcomes need explicit imbalance handling, such as resampling or class weights."],"supporting_citations":[{"why":"Supplies the all-MiniLM-L6-v2 transformer embedding model that encodes essays into the 384-dimensional semantic vectors used by the EQI module.","marker":"[17]"},{"why":"Provides the GPT-4o LLM that produces the rubric scores, prompt alignment scores, extracurricular scores, and SHAP-based feedback text.","marker":"[18]"},{"why":"Introduces SHAP, the feature attribution method used to make EQI predictions interpretable and to identify the dominant rubric-score features.","marker":"[13]"},{"why":"Prior interpretable deep learning approach to undergraduate admission prediction that CAPS extends from validation to holistic review prediction.","marker":"[3]"},{"why":"A learned admission-prediction model that replaces standardized tests, used as context for fairness-aware machine learning in admissions.","marker":"[5]"},{"why":"Provides the college admission counseling association's expert-defined factors that ground the manual weights in the SAS and fusion modules.","marker":"[15]"},{"why":"Supplies the tier-based framework for extracurricular activities that underlies the EIS module's expert evaluation component.","marker":"[14]"}],"fun_headline_variants":["LLM + XGBoost split admissions into 3 scores, hit 75% tier accuracy","CAPS: one explainable score, three components, 80% essay R²","Holistic review decoded: academics, essays, activities as one score","Multi-modal CAPS predicts admissions tiers with 75% accuracy","Interpretable admissions score from grades, essays, activities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic applicant scores and admission tiers faithfully resemble real holistic admissions decisions, because if they do not, the reported accuracy and R² measure the data-generation rules rather than the framework's ability to predict human judgments.","fun_headline_variants_meta":{"raw":{"variants":["LLM + XGBoost split admissions into 3 scores, hit 75% tier accuracy","CAPS: one explainable score, three components, 80% essay R²","Holistic review decoded: academics, essays, activities as one score","Multi-modal CAPS predicts admissions tiers with 75% accuracy","Interpretable admissions score from grades, essays, activities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3196,"prompt_tokens":923,"completion_tokens":2273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2175}},"tokens_in":539,"tokens_out":2273,"duration_ms":18975,"temperature":1.0,"reasoning_tokens":2175,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:57:53.044418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the pipeline a corpus of real application essays with independent human admissions-officer ratings, build the same 387-dimensional features, and test whether the XGBoost regressor still reaches an R² near 0.80 for EQI; then apply the full CAPS score to a university's historical admit/deny records and compare predicted tiers with actual decisions. A large drop in R² or near-chance tier accuracy would show the synthetic results were artifacts of the generation process.","supporting_citations":[{"cited_title":"Admission Prediction in Undergraduate Applications: an Interpretable Deep Learning Approach","cited_arxiv_id":"2401.11698","evidence_quote":"Prior interpretable deep learning approach to undergraduate admission prediction that CAPS extends from validation to holistic review prediction."},{"cited_title":"Factors in the Admission Decision,","cited_arxiv_id":null,"evidence_quote":"Provides the college admission counseling association's expert-defined factors that ground the manual weights in the SAS and fusion modules."},{"cited_title":"Extracurricular involvement in high school is not a level playing field,","cited_arxiv_id":null,"evidence_quote":"Supplies the tier-based framework for extracurricular activities that underlies the EIS module's expert evaluation component."}],"review_version":1}