{"id":"4b19aed4-75a1-40d7-be5f-5114884518e1","arxiv_id":"2509.04461","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage LLM pipeline with retrieval-augmented prompting and synthetic minority oversampling claims state-of-the-art MBTI classification on PersonalityCafe, though its AUC uses neighbor-label distribution, not model output.","lead":"This paper combines a local language model, a similarity search over past forum posts, and a commercial chat model to predict a user's Myers-Briggs type from their social media posts. The authors report large gains over ten older baselines, but the headline AUC is calculated from the retrieved neighbors' true labels rather than from the model's own predictions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test posts contain explicit MBTI self-reports, so P2P's high performance may reflect label extraction rather than personality inference.","rationale":"The reader identified the AUC approximation as the weakest point. While that is a real methodological flaw, it only undermines one of the three headline metrics. The self-report leakage is more fundamental: it calls into question the validity of all metrics and the central task itself. The case study directly shows posts containing 'I was an INTJ' and 'I'm ... ESTP', and the dataset is a well-known MBTI discussion forum where such statements are common. The preprocessing (§4.1) does not filter these out, and the prompt includes the original posts. An LLM can trivially extract these explicit statements, making the task closer to text retrieval than to personality inference. This also explains the unusually high accuracy (~0.93) and F1. The AUC issue is subordinate: even if AUC were computed properly, the accuracy/F1 would still reflect the leakage. Therefore, the most load-bearing concern is the contamination of test inputs with the target labels, and the paper's SOTA claim is not adequately supported as stated.","tokens_in":16002,"tokens_out":10155,"duration_ms":110006,"concrete_test":"Mask or remove all occurrences of the 16 MBTI type strings (e.g., 'INTJ', 'INFP', 'ENTP', etc.) and variants like 'I am an ...' from the test posts, then rerun the full P2P pipeline and the top baselines. If P2P's accuracy and F1 drop substantially (e.g., accuracy from ~0.93 to near baseline levels), it confirms that P2P's performance relies on explicit self-reports rather than genuine personality inference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the evaluation task is contaminated by explicit self-reports of MBTI type in the user posts. The dataset is from PersonalityCafe, a forum dedicated to MBTI discussion, where users frequently write 'I am an INTJ', 'I'm an ENTP', etc. The paper's preprocessing (§4.1) does not strip MBTI type names, and the online LLM in Step 3 receives the original posts in <CONTENT> (prompt in Appendix C). The case studies in Appendix D confirm this: Case 1 contains 'I'm serious considering whether I'm actually an ESTP', and Case 2 contains '...before I knew I was an INTJ'. Thus the input text often contains the ground-truth label itself. P2P's high accuracy (e.g., 0.9321 I/E) may reflect the LLM's ability to read and extract these explicit statements rather than infer personality from indirect linguistic cues. This inflates all three metrics and makes the comparison to 10 baselines unfair, since LLMs are particularly adept at following the instruction 'only output four letters' and copying the answer from the text, while classical features may not recognize such statements. If this is true, the central claim of state-of-the-art personality prediction is not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PostToPersonality (P2P), a dual-LLM framework for MBTI prediction from social media posts. P2P has three stages: a fine-tuned local LLM (DeepSeek-R1-8B) extracts interpretable psychological features; these features are combined with the original posts to retrieve the top-k most similar training entries from a vector database; and an online LLM (DeepSeek-V3) makes the final four-dimensional MBTI prediction using the original posts, the extracted features, and the retrieved demonstrations as in-context examples. The authors claim state-of-the-art performance on the PersonalityCafe dataset, reporting average gains of 8.2% in accuracy, 20.17% in F1, and 4.1% in AUC over 10 ML/DL baselines, with ablations for PEFT, SMOTE, RAG, and model choices.","tokens_in":16343,"tokens_out":4626,"duration_ms":52305,"significance":"If the reported results were trustworthy, P2P would be a meaningful step for MBTI inference from text and would demonstrate a useful integration of fine-tuned local LLMs, retrieval augmentation, and commercial LLM reasoning. The architecture is clearly described and the ablation study is reasonably comprehensive. However, the empirical contribution is currently undermined by two load-bearing problems: the test set appears to contain explicit MBTI self-reports that the online LLM can directly read, and the reported AUC is not computed from the LLM's own predictions but from the label distribution of retrieved neighbors. These issues affect every quantitative claim in the paper, so the central SOTA conclusion is not supported by the present evidence.","major_comments":[{"comment":"The evaluation is contaminated by explicit MBTI self-reports. The preprocessing described in §4.1 does not remove MBTI type names, and the prompt in Appendix C feeds the original posts in <CONTENT> to the online LLM. The dataset is from PersonalityCafe, a forum dedicated to MBTI discussion, and the case studies in Appendix D confirm that users write statements such as \"I'm serious considering whether I'm actually an ESTP\" and \"before I knew I was an INTJ\". Under these conditions, high accuracy (e.g., 0.9321 on I/E) may reflect the LLM extracting a directly stated label rather than inferring personality from indirect linguistic cues. This makes the comparison with the 10 baselines unfair and invalidates the SOTA claim. The authors must quantify how many test posts contain explicit type mentions, remove such posts (or at least report performance on a filtered subset), and rerun all baselin","section":"§4.1 (Dataset) and Appendix C/D"},{"comment":"The AUC reported for P2P is not the AUC of the online LLM. The paper states that \"predictive probabilities for computing AUC are approximated by the retrieved k samples in the RAG\" and that the probability is the distribution of ground-truth labels among those samples. This measures a k-NN classifier on the retrieval database, not the LLM's ranking ability. All baselines compute AUC from their own probability outputs, so the claimed 4.1% average AUC improvement is not a comparison of the same quantity. The paper should report AUC from the LLM's predictive probabilities (e.g., token logits) or omit AUC from the comparison.","section":"§4.1 (Evaluation Metrics)"},{"comment":"All results are single-run point estimates with no error bars, confidence intervals, or significance tests. Since the pipeline uses stochastic API decoding and the retrieval-based approximation, differences such as 0.9306 vs. 0.9321 between rows (a) and (m) of Table 2 could easily be noise. The authors should report mean and standard deviation over multiple runs and, where appropriate, paired significance tests to support the claimed improvements.","section":"Tables 1 and 2"},{"comment":"The DeepSeek-R1-8B baseline appears to be the unfine-tuned local LLM. Its near-chance accuracy on several dimensions (e.g., 0.4181 on N/S, 0.5401 on T/F) suggests that the comparison conflates the effect of fine-tuning with the effect of the full P2P framework. If the baseline is zero-shot, this should be stated explicitly; a stronger baseline would be the same local LLM fine-tuned without RAG, or an equivalently sized fine-tuned LLM.","section":"§4.1 (Baselines) and Table 1"},{"comment":"The SMOTE update is written as x̂_i = x_i + λ(x_i − x_j), λ ∼ Uniform(0,1). This is not the standard SMOTE interpolation x_i + λ(x_j − x_i), and it is inconsistent with the preceding sentence that says P2P interpolates \"between x_i and x_j\". As written, the new sample moves away from x_j rather than along the segment connecting the two points. Please correct the sign or clarify the intended operation and verify the experiments use the corrected formula.","section":"Eq. (2)"}],"minor_comments":[{"comment":"The text in Figure 1 is partially garbled in the manuscript image (e.g., \"ENTJESFP……\"), making the pipeline diagram hard to read. Please provide a clean version.","section":"Figure 1"},{"comment":"The symbols x_i and x_i are used inconsistently for the hidden representation and its average pooling. The same symbol k is used for the number of SMOTE neighbors in Eq. (3) and for the RAG retrieval size in §3.1; please disambiguate.","section":"Notation, Eq. (2)-(3)"},{"comment":"Variant (d) in Table 2 shows a very large drop in N/S accuracy (0.4181) compared with variant (c), but this is not discussed. A short explanation would help readers understand whether this is an instability or an artifact of the setting.","section":"§4.2.2"}],"recommendation":"reject","confidential_remarks":"The test-label contamination is the decisive issue. Because the corpus is an MBTI discussion forum, explicit type statements are likely pervasive, and the current evaluation cannot separate label extraction from personality inference. Unlike a missing significance test or a metric fix, this cannot be repaired by a modest reanalysis of the existing tables; it requires re-collecting or substantially filtering the data and re-running the entire experiment. In my view, that goes beyond a revision and warrants rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the headline numbers don't survive contact with the data. The test posts come from PersonalityCafe, a forum where people discuss MBTI, and the paper's own case studies show users writing 'I'm serious considering whether I'm actually an ESTP' and 'before I knew I was an INTJ'. The prompt feeds the original posts to the online LLM, so the model can literally read the label. The 93% accuracy on I/E is more likely label extraction than personality inference. That's the load-bearing problem; the AUC approximation is a second, smaller one.\n\nWhat's genuinely new: the dual-LLM design (local fine-tuned feature extractor + online RAG prediction) is a reasonable engineering combination, and the ablations show each piece adds something. QA-LoRA with SMOTE on hidden embeddings is a sensible way to handle imbalance. The pipeline is described clearly. No code or data, but the method section is reproducible in principle.\n\nThe soft spots: (1) contamination, as above. Preprocessing removes stopwords but not MBTI type names, and the case-study prompts in Appendix D confirm the leakage. This alone invalidates the SOTA claim. (2) The AUC is computed from the distribution of labels among retrieved neighbors, not from the LLM's own probabilities. That measures a k-NN classifier, so the 4.1% AUC gain over baselines is not an apples-to-apples comparison. (3) No error bars or multiple runs, and DeepSeek-R1-8B is an odd weak baseline—uninstructed base model, likely not representative.\n\nI don't think the authors are being deceptive; the paper is transparent about the AUC approximation, and the case studies are included. But the oversight is fundamental. The framework might work on cleaner data, and a re-evaluation with explicit self-reports stripped (or a dataset not built around MBTI discussion) would be worth seeing. As it stands, the empirical contribution does not support the abstract's claims.\n\nWho should read it: anyone working on psychosocial label inference from social media, as a warning about task contamination. It deserves a serious referee—not a desk reject—because the methodological error is instructive and the framework is non-trivially engineered. But I would not accept it in current form.","headline":"The empirical claims are invalidated by test-set contamination: the forum posts contain explicit MBTI self-reports, so the LLM is largely reading the answer rather than inferring it.","tokens_in":16756,"tokens_out":3039,"would_cite":false,"duration_ms":34632,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a two-LLM retrieval pipeline can predict MBTI personality types from social media posts, beating classical and BERT classifiers by wide margins.","keywords":["MBTI prediction","personality prediction","large language models","retrieval-augmented generation","in-context learning","SMOTE oversampling","social media text","class imbalance"],"falsifier":"Compute P2P's AUC using the online LLM's own predictive probabilities (for example, the log-probabilities of the four output letters) on the same test split and compare with baseline AUCs; if the gap vanishes or reverses, the claimed AUC superiority is an artifact of the retrieval-label proxy. Alternatively, run a k-NN-only predictor on the same vector database without any LLM and check whether it matches P2P's accuracy; if it does, the LLM stages add little beyond retrieval.","tokens_in":15963,"feed_emoji":"🧠","tokens_out":8726,"duration_ms":88263,"temperature":0.7,"pith_summary":"The paper tries to establish that a two-stage large-language-model pipeline can predict a social media user's Myers-Briggs type from their posts more accurately than classical machine learning and fine-tuned BERT baselines. The proposed system, PostToPersonality (P2P), first has a locally fine-tuned LLM turn a user's posts into a psychological assessment, then retrieves similar post-label examples from a vector database, and finally asks an online LLM to make the four binary MBTI judgments with those examples as in-context demonstrations. To handle the population's imbalanced MBTI distribution, the local model is fine-tuned with synthetic minority oversampling in embedding space. On a public forum dataset of 8,675 users, P2P reports average improvements of 8.2% in accuracy, 20.17% in F1, and 4.1% in AUC over 10 baselines. If correct, this would make LLM-based retrieval-augmented inference a practical step beyond feature-engineered classifiers for personality prediction from text.","feed_headline":"Dual-LLM pipeline hits 93% accuracy on MBTI, beats ten baselines","feed_subtitle":"Retrieval-augmented generation plus minority oversampling outperforms classical ML and BERT on all four MBTI axes.","key_machinery":"The load-bearing mechanism is a retrieval-augmented dual-LLM loop. Step one: a locally deployed LLM, fine-tuned on the MBTI prediction task with synthetic minority oversampling, converts raw posts into an interpretable assessment of the user's social tendency, information processing, decision-making, and lifestyle. Step two: the assessment and the posts are embedded and concatenated into a query vector, and exact k-nearest-neighbor search over a vector database of training users returns the k most similar posts together with their ground-truth MBTI labels. Step three: the original posts, the local assessment, and the retrieved demonstrations are assembled into one prompt for an online LLM, w","core_discovery":"The central claim is that LLM-based MBTI prediction fails when a single model is asked to reason directly from raw posts, and that the failure can be repaired by splitting the task across two models connected by a retrieval index. The paper's P2P framework first runs a locally fine-tuned LLM over a user's posts to produce a short psychological feature text. Those features, together with the posts, are embedded and used to retrieve the top-k most similar training users from a vector database; the retrieved posts and their ground-truth MBTI labels become demonstrations in a prompt sent to an online LLM, which returns the four-letter type. The paper reports that this design raises average accur","pith_inferences":["Inference: A stricter evaluation would compute AUC from the online LLM's own predicted probabilities rather than from the retrieved neighbors' label distribution; that test could narrow the reported 4.1% AUC margin.","Inference: The same two-stage retrieval recipe should transfer to other text-based psychological inventories such as the Big Five, since nothing in the design is MBTI-specific.","Inference: Because retrieval alone carries much of the gain, a k-nearest-neighbor classifier on the same vector database would reveal how much of P2P's accuracy comes from the LLM stages versus the index itself.","Inference: Synthetic oversampling in embedding space creates training points that have no textual form; generating synthetic posts and re-embedding them might improve minority-type robustness further."],"forward_implications":["Removing RAG drops I/E accuracy from 0.9321 to 0.8287, so retrieved demonstrations are the largest single contributor to P2P's performance.","Fine-tuning without SMOTE still beats no fine-tuning, but SMOTE recovers accuracy on minority MBTI types such as ESFJ and ESTJ.","Swapping the online LLM to a different family lowers accuracy, suggesting that model-family consistency between the feature extractor and the final predictor matters.","All models, including P2P, score lowest on judging versus perceiving, marking that dimension as the hardest to infer from text.","The full pipeline's gains over BERT-style encoders support the use of dual-LLM retrieval-augmented inference for personality-related text classification."],"supporting_citations":[{"why":"Supplies the social-media forum dataset of users' posts with MBTI ground-truth labels used for training and testing.","marker":"[4]"},{"why":"SMOTE is the synthetic minority oversampling technique whose embedding-space interpolation is adapted for fine-tuning the local LLM.","marker":"[9]"},{"why":"XGBoost is one of the strongest classical baselines that P2P's reported gains are measured against.","marker":"[10]"},{"why":"The local reasoning LLM is the fine-tuned model that generates psychological feature text for retrieval and prompting.","marker":"[14]"},{"why":"In-context learning is the mechanism by which retrieved post-label demonstrations steer the online LLM's final prediction.","marker":"[16]"},{"why":"Supplies the exact k-nearest-neighbor vector search used to retrieve similar users and their MBTI labels.","marker":"[22]"},{"why":"Produces the post embeddings that, combined with local-LLM feature embeddings, form the retrieval query and database vectors.","marker":"[31]"},{"why":"Provides the parameter-efficient fine-tuning algorithm used to adapt the local LLM to the MBTI task.","marker":"[38]"}],"fun_headline_variants":["Two-step LLM: fine-tune features, then retrieval-augment for MBTI","Dual LLM + RAG: 93% MBTI, outperforms 10 baselines","For MBTI from posts: one LLM extracts, another retrieves and answers","LLM duo fixes hallucination and imbalance for 93% MBTI accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the ranking-quality metric is faithfully estimated from the ground-truth labels of the retrieved neighbor posts; if the online LLM's own confidence order differs from that neighbor-label distribution, the reported advantage over baselines is not established.","fun_headline_variants_meta":{"raw":{"variants":["Two-step LLM: fine-tune features, then retrieval-augment for MBTI","Dual LLM + RAG: 93% MBTI, outperforms 10 baselines","For MBTI from posts: one LLM extracts, another retrieves and answers","LLM duo fixes hallucination and imbalance for 93% MBTI accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2337,"prompt_tokens":748,"completion_tokens":1589,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1497}},"tokens_in":492,"tokens_out":1589,"duration_ms":14391,"temperature":1.0,"reasoning_tokens":1497,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:04:38.076253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute P2P's AUC using the online LLM's own predictive probabilities (for example, the log-probabilities of the four output letters) on the same test split and compare with baseline AUCs; if the gap vanishes or reverses, the claimed AUC superiority is an artifact of the retrieval-label proxy. Alternatively, run a k-NN-only predictor on the same vector database without any LLM and check whether it matches P2P's accuracy; if it does, the LLM stages add little beyond retrieval.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the social-media forum dataset of users' posts with MBTI ground-truth labels used for training and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the parameter-efficient fine-tuning algorithm used to adapt the local LLM to the MBTI task."}],"review_version":1}