{"id":"d7dadb8c-428f-4e70-ba05-f7074c0a5762","arxiv_id":"2505.14725","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HR-VILAGE-3K3M harmonizes bulk and single-cell transcriptomic profiles from 66 studies into one openly hosted resource for studying human respiratory viral immunization.","lead":"This paper introduces HR-VILAGE-3K3M, a curated collection of gene expression data from 3,178 people in 66 vaccination and viral exposure studies. It aims to give researchers a single, standardized resource for studying how the immune system responds to respiratory viruses over time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The repository's central utility hinges on subject-level outcome labels and ID mappings that the paper does not audit; an undetected mapping error would propagate through all benchmark results.","rationale":"The reader's weakest assumption correctly identifies metadata and outcome-label correctness as the soft spot. I agree the dataset existence claim is plausible and the curation effort described in Sections 1 and 2 is substantial; the naming inconsistencies and unsupported 'largest' wording are real but secondary. I do not see an internal mathematical contradiction in the preprocessing pipeline, and the machine-actionable Hugging Face and GitHub links, together with the concrete benchmark descriptions, give the paper independent support beyond a purely theoretical claim. My stress-test concern is that the most advertised downstream capability—response prediction—is only as sound as the manually reconciled responder labels, and the paper provides no audit of that reconciliation. Because the remedy is a conditional requirement rather than a demonstrated failure, the verdict should remain conditional; no change from the reader's verdict is needed. I mark agreement as partial because the reader emphasizes label correctness, whereas I would also require an artifact-level count and link verification as part of the same condition.","tokens_in":22271,"tokens_out":6670,"duration_ms":67530,"concrete_test":"Download the Hugging Face dataset and reconstruct the exact benchmark set of Figure 2. For each of the 20 influenza vaccine studies, obtain the original antibody files from GEO/ImmPort/supplementary materials, map each expression-profile accession row to a subject ID using only the original deposited sample-to-subject tables, and recompute high/non-responder labels with the paper's MFC>=4/Day-28>=40 criterion. Then rerun the PCA-logistic and Transformer pipelines with the paper's hyperparameters on the relabeled cohort, comparing the resulting accuracy, AUC, and F1 against Figure 2. If subject-level label disagreement exceeds 2% of benchmark participants, or if either model's AUC or F1 shifts by more than 0.05, the reported benchmark results and the 'rigorously curated' claim require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that HR-VILAGE-3K3M is a rigorously curated, AI-ready resource with 3,178 subjects, 66 studies, and 14,136 profiles—has two unverified load-bearing components. First, the existence and exact contents of the hosted artifact are asserted via URL only; no manifest, count table, checksums, or row-level consistency check is included in the paper. Second, every benchmark in Section 3 uses responder labels built as described in Section 2 by manually resolving mismatched sample IDs and e-mailing study authors for missing antibody data. The paper itself states in Section 1 that metadata 'use sample IDs that differ from those used by the expression data in several instances, making data merging impossible' without significant human effort, yet no error rate, reconciliation log, or independent audit of those links is reported. If even a few percent of subject-to-expression links or responder classifications are wrong, the 'balanced' 20-study/1,268-participant cohort in Section 3 can be systematically mislabeled, and the reported AUC/F1 differences among QN, ComBat, and Regression (Figure 2) may measure label artifacts rather than biological signal. Additionally, the responder definition (MFC>=4 and Day-28 HAI>=40 for high, MFC<=1 for non-responders) is applied across studies that may not share the same assay or day-28 measurement, and the paper does not document how such discrepancies were harmonized. This is an internal reliability condition for the resource's main advertised use case, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HR-VILAGE-3K3M, a curated longitudinal transcriptomic repository for human respiratory viral immunization studies, integrating 14,136 RNA-seq profiles from 3,178 subjects across 66 studies, with additional single-cell data reported as over 2.56 million cells. The authors describe a detailed curation workflow: collection from GEO, ImmPort, and ArrayExpress; harmonization of gene symbols; resolution of duplicate samples; standardized preprocessing for microarray, bulk RNA-seq, and scRNA-seq; and definition of responder status for influenza vaccine studies based on HAI titers. To demonstrate utility, they report two analyses: a benchmark of batch-effect correction methods and deep-learning sequence models for antibody responder prediction on 20 influenza vaccine studies, and a paired bulk/scRNA-seq cell-type annotation analysis of SARS-CoV-2 vaccine samples. The paper's central claim is that this resource is an AI-ready, rigorously curated benchmark for systems immunology, with data and code publicly hosted.","tokens_in":22589,"tokens_out":4371,"duration_ms":42749,"significance":"If the hosted dataset is complete and correctly curated, HR-VILAGE-3K3M would be a substantial and useful integrative resource for systems immunology, filling a gap in harmonized longitudinal transcriptomic data for respiratory viral immunization. The paper's strengths include the open availability of data and code, the explicit attempt to standardize outcome definitions across studies, the reporting of preprocessing details, and the inclusion of paired bulk and single-cell analyses. The benchmark analyses, while internal, provide a reasonable illustration of possible downstream uses. However, the central claims cannot be fully assessed from the manuscript alone because the hosted artifact is not independently auditable from the text: the paper lacks a manifest or per-study count table, and the subject-level outcome labels that underpin all benchmarks rest on ID-mapping and antibody-curation steps whose error rates are not reported. These are fixable with additional documentation and validation, but they are load-bearing for the paper's main contribution.","major_comments":[{"comment":"The aggregate counts (3,178 subjects, 14,136 profiles, 66 studies) and the abstract's 'over 2.56 million cells' are the paper's central quantitative claims, but the manuscript provides no per-study count table, no manifest of the hosted files, and no checksums or scripts that regenerate these totals from the Hugging Face release. Because the dataset itself is the contribution, please add a machine-readable manifest with per-study accessions, platform, subject/sample/cell counts, and a reproducibility script that recomputes the aggregate numbers from the released files.","section":"Section 2, Summary (and Appendix A)"},{"comment":"The responder labels used in all Section 3 benchmarks are defined as MFC≥4 with Day-28 HAI≥40 (high responders) and MFC≤1 (non-responders), but the paper does not document how each of the 20 benchmark studies satisfies the Day-28 measurement requirement or how assay differences (e.g., hemagglutination inhibition assay variants, egg- versus cell-based antigens, different laboratories) were harmonized. Without a per-study endpoint table specifying the exact assay, the measurement day, and the numbers reclassified from original study definitions, the cross-study responder labels and the reported AUC/F1 differences are not auditable.","section":"Section 2, Outcome definition"},{"comment":"The paper states that metadata sample IDs differ from expression-data IDs in several instances and that merging required significant human effort, including contacting study authors, yet no reconciliation log, error count, or independent validation of subject-to-sample and subject-to-antibody links is reported. Since every benchmark in Section 3 depends on these links, an undetected ID mapping error would propagate directly into the responder prediction results. Please release the mapping tables (or appropriately de-identified versions) and describe validation checks, such as sex/age consistency, duplicate-sample Spearman verification, and any resulting exclusion rates.","section":"Section 1 and Section 2, Data quality control"},{"comment":"The benchmark cohort of 20 influenza vaccine studies and 1,268 participants is selected to be 'balanced' and to exclude studies with 'extremely unbalanced' responder distributions, but the exact selection criteria and per-study responder counts are not reported. Moreover, the modeling results are evaluated internally on the same resource and do not constitute external validation. Please provide the full inclusion/exclusion audit trail, per-study responder counts, and explicit statements framing these analyses as internal utility demonstrations rather than validated predictive performance.","section":"Section 3, Outcome Prediction Using Longitudinal Bulk Data"},{"comment":"The conclusion that QN 'consistently outperformed' ComBat and Regression is based on mean metrics averaged over six random seeds, but Figure 2b does not show error bars, standard deviations, or statistical tests. Because the reported differences (e.g., Transformer AUC 0.727 vs. 0.669) may be within seed-level variability, please provide per-seed results, confidence intervals, or paired significance tests before asserting consistent superiority of one normalization method.","section":"Section 3, Figure 2b"}],"minor_comments":[{"comment":"The name 'HV-RIGEL-3K3M' appears in the Figure 2 caption and the Section 4 heading, whereas the dataset is consistently called 'HR-VILAGE-3K3M' elsewhere; please unify the spelling.","section":"Figure 2 caption and Section 4 title"},{"comment":"The main text refers to 'section D', 'section E', and 'section F', but the appendix sections are not explicitly numbered; please number them or use descriptive cross-references.","section":"Appendix D, E, F"},{"comment":"The abstract reports 'over 2.56 million cells', but Section 2 does not provide per-study cell counts or define which seven single-cell studies contribute to this total; please add a per-study cell count summary.","section":"Section 2, Summary and abstract"},{"comment":"Appendix B lists reference numbers without a mapping between GEO/ImmPort accessions and the corresponding dataset references; a table linking accession, reference, platform, and inclusion status would greatly improve traceability.","section":"Appendix B"},{"comment":"The single-cell annotation details mention GSE201534 and GSE246937, but GSE246937 does not appear elsewhere in the main text; please ensure the accessions are consistent and cross-referenced.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a data descriptor whose central contribution is a hosted repository, but I was not able to inspect the hosted artifact itself. My assessment is therefore conditional on the authors supplying the audit materials requested in the major comments: a machine-readable manifest, per-study outcome and assay documentation, reconciliation logs, and per-seed benchmark results. If those materials confirm the reported counts and label integrity, the resource is likely a valuable community asset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the integrated resource itself, not the benchmark numbers. If the ID mapping and outcome labels are right, this is a real contribution to systems immunology. What's actually new is the scale and harmonization: 66 studies, 3,178 subjects, 14,136 profiles, plus single-cell data, with unified metadata, outcome definitions, and preprocessing. That extends the authors' earlier collection, but it's a meaningful step beyond it. The preprocessing is described in unusual detail—RMA, QN, HGNC symbol correction, duplicate removal via Spearman correlation, explicit responder thresholds. The counts add up, and the Hugging Face hosting plus GitHub code are good practice.\n\nThe soft spots are exactly where the stress-test note points. The load-bearing parts are the subject-to-sample links and responder labels. Section 1 admits sample IDs sometimes differ from expression IDs and that antibody data had to be obtained by emailing authors, but the paper gives no reconciliation log, no error rate, no independent audit. If even a few percent of those links are wrong, the Section 3 benchmarks could be measuring label artifacts rather than biological signal. That is not a fatal flaw in the resource, but it is a gap in the evidence for the word \"rigorous.\" Relatedly, the responder definition (MFC>=4, day-28 HAI>=40) is applied across studies that may not share assays or a day-28 measurement; the paper doesn't document how such discrepancies were harmonized. The benchmark also excludes unbalanced studies by design, so the reported AUC/F1 numbers are conditional on that filtering. Minor issues: the \"largest\" claim is unsubstantiated, and Figure 2's caption says \"HV-RIGEL\" instead of \"HR-VILAGE.\"\n\nNone of this sinks the paper. The curation effort is real, the resource fills a gap, and the limitations the authors do acknowledge are honest. The missing audit trail is fixable with a manifest, a reconciliation log, and sensitivity analyses around the outcome definitions. I'd want those before trusting the benchmark comparisons, but the dataset itself deserves to be evaluated by a serious referee.\n\nWho this is for: people building integrated transcriptomic resources, and anyone doing vaccine-response prediction or batch-effect benchmarks. I'd bring it to a reading group focused on data curation rather than methods. My recommendation: send it to peer review, but the reviewers should ask for the audit and harmonization documentation before acceptance.","headline":"A substantial curated resource that is probably useful, but the paper's central promise depends on subject-level labels and ID links that aren't audited; worth serious review.","tokens_in":23141,"tokens_out":1583,"would_cite":true,"duration_ms":15531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces HR-VILAGE-3K3M, a publicly available, AI-ready dataset that unifies 14,136 longitudinal RNA-seq profiles from 3,178 subjects across 66 respiratory-viral vaccination and inoculation studies.","keywords":["HR-VILAGE-3K3M","systems immunology","longitudinal transcriptomics","vaccine response prediction","data harmonization","batch effect correction","single-cell RNA-seq","benchmark dataset"],"falsifier":"Take a random sample of studies in HR-VILAGE-3K3M, retrieve the original supplementary clinical tables and any author-provided antibody files, and independently reconstruct subject-to-sample links and responder labels; if mismatches are found at a non-negligible rate, the outcome labels used in Section 3 are unreliable. A quick quantitative check is to rerun the responder-prediction benchmark excluding all studies whose antibody data had to be obtained by email and see whether the reported F1 and AUC hold.","tokens_in":22099,"feed_emoji":"🧬","tokens_out":7513,"duration_ms":66633,"temperature":0.7,"pith_summary":"This paper introduces HR-VILAGE-3K3M, a public, AI-ready dataset built by merging 14,136 longitudinal RNA-seq profiles from 3,178 subjects in 66 studies of respiratory viral vaccination and controlled infection. The authors' central claim is that after harmonizing gene symbols, sample identifiers, subject metadata, and outcome definitions, and after removing duplicate and technically corrupted samples, these scattered public datasets form one dependable resource for systems-immunology research. The value of the claim, if correct, is that researchers can skip the time-consuming step of data cleaning and ID matching and go directly to training and benchmarking predictive models, testing batch-effect corrections, and studying immune trajectories over time. The paper also demonstrates the resource with two analyses: predicting antibody responder status after influenza vaccination with machine-learning models, and comparing bulk and single-cell estimates of immune cell-type changes after COVID-19 vaccination.","feed_headline":"One curated repository spans 3,178 people and 66 immunization studies","feed_subtitle":"Harmonized sample IDs, gene symbols, and responder labels make the data ready for machine-learning benchmarks.","key_machinery":"The load-bearing mechanism is the curation and harmonization pipeline rather than any single statistical model. Its parts are: subject-level metadata reconstruction, including resolving sample-ID mismatches and emailing study authors for missing antibody titers; standardized preprocessing, with RMA and quantile normalization for microarray data, read alignment and count generation for RNA-seq, and total-count-plus-log1p normalization for single-cell data; gene-symbol correction to official HGNC names, using a tool that repairs invalid or outdated symbols; duplicate-sample detection through protocol review and Spearman correlation of expression profiles; and standardized outcome definitions, notably the HAI-titer maximum-fold-change thresholds for influenza vaccine response. These steps convert raw, scattered depositions into expression matrices whose row names align with metadata and whose outcome labels are comparable across studies; every downstream example analysis depends on this alignment.","core_discovery":"The core claim is infrastructural: that a carefully curated integration of existing public transcriptomic studies can serve as a standardized benchmark for the human immune response to respiratory viral immunization. Specifically, the authors report assembling 3,178 participants—462 from 15 inoculation studies covering H3N2, H1N1, HRV, RSV, and SARS-CoV-2, 2,412 from 47 influenza and COVID-19 vaccination studies, and 304 from four mixed-exposure studies—profiled at up to 22 time points across whole blood, PBMCs, and nasal swabs, using microarray, bulk RNA-seq, and 10x single-cell platforms. They standardize gene symbols to official HGNC names, assign unique GEO accession identifiers to samples, define a consistent subject identifier, and set common outcome criteria: for influenza vaccines, high responders are those with maximum HAI fold change at least 4 and a Day-28 titer at least 40; non-responders have fold change at most 1; COVID-19 vaccine studies carry raw antibody data so users can choose their own thresholds. In their benchmark on 20 influenza vaccine studies with 1,268 participants, PCA-based logistic regression after quantile normalization achieved F1 0.822, AUC 0.748, and accuracy 0.726, surpassing RNN, LSTM, GRU, and Transformer models, with the QN-corrected Transformer the best deep model (F1 0.804). The paired bulk and single-cell demonstration for two SARS-CoV-2 vaccine studies shows both modalities detect a rise in B cells from Day 1 to Day 7 and a fall in monocytes and NK cells.","pith_inferences":["Editorial inference: if the dataset becomes a standard benchmark, the paper's binary responder definition (MFC at least 4 and titer at least 40) may harden into a default even though antibody responses are continuous; using a threshold discards information about moderate responders.","Editorial inference: the strongest test of the curation is reproducibility of outcome labels; an independent re-derivation of antibody titers from original study supplements and author correspondence would either confirm or undermine every supervised benchmark in Section 3.","Editorial inference: because the benchmark found a simple PCA-logistic model beating all deep sequence models, a likely downstream effect is that researchers will treat quantile normalization plus logistic regression as a strong baseline that any longitudinal deep-learning claim must surpass on this resource.","Editorial inference: the paired bulk and single-cell datasets cover PBMCs from COVID-19 vaccine studies; extending the same paired design to nasal swabs would test whether local mucosal immune dynamics track systemic blood dynamics, a question the current resource cannot answer."],"forward_implications":["Researchers can train and compare models for early prediction of vaccine response on a common set of 20 influenza studies with harmonized responder labels; the paper's own benchmark shows quantile normalization giving better results than ComBat or regression batch correction.","The paired bulk and single-cell datasets from the same subjects give deconvolution methods a check: cell proportions from scRNA-seq can be compared with enrichment scores from bulk RNA-seq at matching time points.","The platform-structured missingness of genes, where availability is determined by study and platform, provides a benchmark for imputation methods that do not assume missing-at-random.","Irregular sampling times and multiple time points per subject make the dataset suitable for change-point detection and causal inference over immune trajectories.","At its stated scale, the dataset is positioned as pretraining material for foundation models of immune response, with downstream fine-tuning for outcome prediction or transfer to smaller studies."],"supporting_citations":[{"why":"Supplies the initial curated collection of blood and nasal transcriptomic datasets and the search strategy this repository extends.","marker":"[24]"},{"why":"The public expression archive that provides the largest share of the source microarray and RNA-seq data.","marker":"[16]"},{"why":"The immunology data portal that supplies many of the expression profiles and antibody measurements.","marker":"[17]"},{"why":"The European array archive that contributes additional RNA-seq datasets.","marker":"[18]"},{"why":"Defines the RMA normalization pipeline used to standardize Affymetrix microarray data.","marker":"[25]"},{"why":"Defines quantile normalization, used both in preprocessing and as a batch-effect benchmark method.","marker":"[26]"},{"why":"Provides the gene-symbol correction used to standardize all genes to official names.","marker":"[28]"},{"why":"Supplies the influenza vaccine responder definition (maximum fold change and Day-28 titer thresholds) used for outcome labels.","marker":"[30]"},{"why":"Provides the SingleR reference-based annotation used for single-cell cell-type labels.","marker":"[39]"},{"why":"Provides the xCell enrichment scores used to compare bulk and single-cell immune cell-type dynamics.","marker":"[41]"}],"fun_headline_variants":["3,178 subjects, 66 studies: one curated immune dataset","AI-ready respiratory viral immunization gene expression resource","Harmonized transcriptomics from 3,178 people for immunity research","Benchmark dataset for respiratory viral immunization responses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire resource depends on the correctness of the manually resolved subject-level metadata and outcome labels, especially antibody titers obtained by emailing study authors and sample IDs matched by hand, since an ID mapping error or a wrong responder label would silently corrupt every expression-to-outcome link and every benchmark built on it.","fun_headline_variants_meta":{"raw":{"variants":["3,178 subjects, 66 studies: one curated immune dataset","AI-ready respiratory viral immunization gene expression resource","Harmonized transcriptomics from 3,178 people for immunity research","Benchmark dataset for respiratory viral immunization responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":2042,"prompt_tokens":1108,"completion_tokens":934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":724,"completion_tokens_details":{"reasoning_tokens":868}},"tokens_in":724,"tokens_out":934,"duration_ms":8013,"temperature":1.0,"reasoning_tokens":868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:40.722111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of studies in HR-VILAGE-3K3M, retrieve the original supplementary clinical tables and any author-provided antibody files, and independently reconstruct subject-to-sample links and responder labels; if mismatches are found at a non-negligible rate, the outcome labels used in Section 3 are unreliable. A quick quantitative check is to rerun the responder-prediction benchmark excluding all studies whose antibody data had to be obtained by email and see whether the reported F1 and AUC hold.","supporting_citations":[{"cited_title":"Hagood, Raymond J","cited_arxiv_id":null,"evidence_quote":"Supplies the initial curated collection of blood and nasal transcriptomic datasets and the search strategy this repository extends."},{"cited_title":"Ncbi geo: archive for functional genomics data sets—update.Nucleic acids research, 41(D1):D991–D995, 2012","cited_arxiv_id":null,"evidence_quote":"The public expression archive that provides the largest share of the source microarray and RNA-seq data."},{"cited_title":"Immport, toward repurposing of open access immunological assay data for translational and clinical research","cited_arxiv_id":null,"evidence_quote":"The immunology data portal that supplies many of the expression profiles and antibody measurements."},{"cited_title":"Arrayexpress—a public database of microarray experiments and gene expression profiles","cited_arxiv_id":null,"evidence_quote":"The European array archive that contributes additional RNA-seq datasets."},{"cited_title":"Summaries of affymetrix genechip probe level data.Nucleic acids research, 31(4):e15–e15, 2003","cited_arxiv_id":null,"evidence_quote":"Defines the RMA normalization pipeline used to standardize Affymetrix microarray data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines quantile normalization, used both in preprocessing and as a batch-effect benchmark method."},{"cited_title":"Hgnchelper: identification and correction of invalid gene symbols for human and mouse.F1000Research, 9:1493, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the gene-symbol correction used to standardize all genes to official names."},{"cited_title":"Multicohort analysis reveals baseline transcriptional predictors of influenza vaccination responses.Science immunology, 2(14):eaal4656, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the influenza vaccine responder definition (maximum fold change and Day-28 titer thresholds) used for outcome labels."},{"cited_title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage.Nature immunology, 20(2):163–172, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the SingleR reference-based annotation used for single-cell cell-type labels."},{"cited_title":"xcell: digitally portraying the tissue cellular heterogeneity landscape.Genome biology, 18:1–14, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the xCell enrichment scores used to compare bulk and single-cell immune cell-type dynamics."}],"review_version":1}