{"id":"782b6c8c-7ed7-46d2-966b-8e0b598d9863","arxiv_id":"2601.21700","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OG-MAR improves LLM prediction of survey responses by retrieving demographically matched World Values Survey profiles and ontology-derived value relations, then aggregating persona-agent answers with a judge agent.","lead":"The paper builds OG-MAR, a pipeline that retrieves demographically similar World Values Survey respondents and ontology-style value relations, then has LLM 'persona' agents answer culture-sensitive questions before a judge agent combines their answers. Across six regional surveys and four LLMs it beats simpler retrieval and debate baselines by a few accuracy points, at the highest token cost of any method tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation-protocol ambiguity: test instances may be k-means centroids, not labeled respondents, and key hyperparameters are chosen on the test benchmarks.","rationale":"The reader's weakest_assumption targets the demographic-proximity retrieval channel, which is an interpretability/mechanism concern: even if demographically similar WVS respondents are imperfect proxies, OG-MAR could still beat baselines by exploiting correlations. The more load-bearing issue is whether the benchmark numbers themselves are valid and fixed. The paper's own appendix introduces two concrete threats: the test points may be synthetic centroids rather than labeled respondents, and the main hyperparameters (K, N, p) are selected on the test benchmarks. These are not internal-mechanism assumptions; they affect every reported accuracy in Table 1. I do not claim the authors intentionally manipulated results — the ambiguity is enough. Because the reader already assigned CONDITIONAL and flagged test-instance selection and tuning (points 2 and 5), my concern reinforces that verdict rather than changing it. The resolution is straightforward if the code/data and explicit sampling protocol are released; until then, the central claim is conditional on the evaluation being real and pre-specified.","tokens_in":32290,"tokens_out":7248,"duration_ms":81152,"concrete_test":"Require the authors to release the exact construction of the 2,000 evaluation instances: for each instance, the original dataset row ID, the gold response, and whether the instance is a centroid or a mapped respondent. Then re-run Table 1 with (i) a random/stratified sample of real respondents rather than centroid prototypes, and (ii) K, N, and p fixed on a validation split before observing test accuracy. If the reported gains over ValuesRAG persist under both changes, the central claim stands; if they shrink or vanish, the comparison is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on accuracy numbers in Table 1, but the evaluation protocol contains a validity gap that is more directly threatening than the demographic-transfer assumption. In Section B.2, the 2,000 evaluation points are described as k-means centroid prototypes (441 AFRO, 74 CGSS, 545 EVS, 75 GSS, 275 ISD, 590 LAPOP) 'used as a compact set of representative points for subsequent analysis.' A centroid in embedding space is not a survey respondent and has no gold label. The paper never specifies whether each centroid is mapped to the nearest real respondent (and if so, how ties or label conflicts are handled) or whether the 'instances' are the centroids themselves. If the latter, the accuracy values in Table 1 are not calculated against real responses. Moreover, Sections 5.1.1 and H.3 select K=5 and N=3 by maximizing test-set accuracy across the six benchmarks, and then Table 1 reports OG-MAR with those tuned settings while baselines use fixed, untuned configurations. The tuning effect is not negligible: Figure 3 shows K=5 vs. K=3 swings of 0.02–0.05 on several backbones, while OG-MAR's average gain over ValuesRAG is as small as +0.005 on GPT-4o-mini. Thus the headline comparison is potentially within the range of selection-on-test bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OG-MAR, an ontology-guided multi-agent framework for culturally aligned LLM inference. The pipeline preprocesses World Values Survey (WVS) respondent data into topic-aware value summaries, builds a cultural value ontology via competency questions and expert review, and at inference retrieves ontology triples and demographically similar WVS profiles to instantiate value-persona agents. A judgment agent synthesizes persona outputs under an evidence-first, ontology- and demographic-aware protocol. The work is evaluated on six regional social-survey benchmarks (EVS, GSS, CGSS, ISD, Afrobarometer, LAPOP) across four LLM backbones, reporting average accuracy gains over zero-shot, role, self-consistency, debate, and ValuesRAG baselines. The paper also includes ablations on the number of retrieved individuals K and ontology triples N, a single-judge comparison, and a human evaluation of reasoning traces.","tokens_in":32508,"tokens_out":2873,"duration_ms":33670,"significance":"If the evaluation is valid, the contribution is useful: structuring value knowledge as an explicit ontology and injecting it through demographically matched persona agents is a plausible and empirically testable mechanism for cultural alignment. The paper has notable strengths: the test labels come from six external surveys independent of the WVS retrieval corpus, the code and demo are promised, the ontology construction uses a fixed taxonomy with expert cross-validation, and the reasoning traces are shown in case studies. The central quantitative claim, however, currently rests on an evaluation protocol whose validity is not established: the test instances appear to be k-means centroids rather than labeled respondents (Appendix B.2), and key hyperparameters K and N are selected on the test benchmarks before final accuracy is reported. Either issue alone would be fixable, but together they make the headline numbers, especially the small GPT-4o-mini gain over ValuesRAG (+0.0046 average), uninterpretable. The paper's significance is real but conditional on correcting these protocol gaps.","major_comments":[{"comment":"The evaluation instances are described as k-means centroids: 'we extract 2,000 centroid prototypes across the six datasets, and use these prototypes as a compact set of representative points for subsequent analysis.' A centroid in embedding space is not a survey respondent and carries no gold label. The manuscript never says whether each centroid is mapped to a nearest real respondent (and how label conflicts are resolved) or whether the centroids themselves are fed to the models for classification. If the latter, the accuracy values in Table 1 are not computed against human survey responses at all. This is a load-bearing validity gap: every headline result depends on these 2,000 instances. Please specify the exact protocol—if centroids are matched to labeled respondents, give the matching rule and its failure/tie rate; if not, the experiments need to be rerun on sampled respondents or a","section":"Appendix B.2 (main results, Table 1)"},{"comment":"Hyperparameters K (number of personas) and N (number of retrieved ontology triples) are selected by maximizing accuracy across the six test benchmarks, and Table 1 reports OG-MAR with those tuned settings. The text states 'we adopt K=5 as the default retrieval depth' after observing the best average in Figure 3, and H.3 selects N=3 similarly. Baselines, by contrast, appear to use fixed, untuned configurations. The tuning effect is not negligible: Figure 3 shows K=5 vs. K=3 differences of 0.02–0.05, while OG-MAR's average advantage over ValuesRAG on GPT-4o-mini is only +0.005 (Table 1). Selecting K and N on the same data used to report final accuracy introduces a selection-on-test bias that can fully absorb such small gains. Please report accuracies with K and N chosen on a held-out validation split, or otherwise demonstrate that the reported comparisons are not the result of test-set tun","section":"Sections 5.1.1 and H.3"},{"comment":"The topic classifier that routes each query into value categories has top-1 accuracy of only 0.34–0.54 on the six test sets, and top-3 accuracy of 0.67–0.80. Since ontology triple retrieval is restricted to the selected categories F_q, a routing error removes the correct value context entirely. This mechanism is central to the method's claimed advantage, yet the paper reports only overall accuracy. Please report OG-MAR's accuracy broken down by whether the true category is in the top-1/top-3 routing predictions, or otherwise quantify how routing errors affect the final predictions. As written, the reader cannot tell whether the gains come from correct ontology grounding or from the agent/judge machinery compensating for misrouting.","section":"Table 3, Section 3.2.1(a)"},{"comment":"The abstract and Section 5 claim 'consistent gains' across four backbones, but the evidence is mixed. On GPT-4o-mini, OG-MAR's average accuracy is 0.6007 vs. ValuesRAG's 0.5961—a difference of 0.0046—and only two of six datasets reach significance (EVS, CGSS under the Holm–Bonferroni-corrected paired t-test). On GSS and ISD, ValuesRAG beats OG-MAR. The discussion in Section 5.1.2 further shows that the Value Inference Variant outperforms OG-MAR on GSS across all four models. The claim of 'strong and consistent performance' should be tempered to reflect the actual margins and significance pattern. Consider reporting effect sizes and per-dataset confidence intervals in addition to the average.","section":"Table 1 and Section 5.1.2"}],"minor_comments":[{"comment":"The main text defines 'M' as the number of retrieved ontology triples (Eq. 9, 'top-M'), while the implementation details and Appendix H.3 call this 'N' ('top-3 triples per category'). Please use one symbol consistently.","section":"Section 3.2.1 vs. H.3"},{"comment":"'GPT-o4-mini' appears to be a typo for 'GPT-4o-mini' or 'o4-mini' as referenced later.","section":"Section 4.1"},{"comment":"The term 'hyper edges' is introduced in the prompts (Appendix E) and case studies, but the framework description consistently calls them 'ontology triples'. Please reconcile the terminology.","section":"Section 3.2.2"},{"comment":"The human evaluation reports mean scores from nine raters but provides no inter-rater reliability (e.g., Krippendorff's alpha or ICC). Given the small rater pool, please report agreement metrics or at least per-rater variance.","section":"Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid idea and a reasonably careful ontology-construction pipeline, but the evaluation protocol contains a potentially fatal ambiguity: if the 2,000 test instances are truly unlabeled centroids, all reported accuracies are meaningless. Given that this is easily clarified, I would not reject outright; the authors need to provide the exact instance-label mapping and rerun the headline experiments with a clean held-out validation split for K and N. The small GPT-4o-mini margin is the case most likely to collapse under proper tuning, but the evidence is not yet sufficient to reject the central claim. I also note that the ontology and the topic classifier are both built from WVS data, so external labels alone do not make all components fully independent; the paper would benefit from at least a discussion of this partial overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing about: OG-MAR is a real system, not a prompt tweak. Combining a CQ-built ontology, demographically retrieved WVS personas, and a constrained judge is a genuine integration, and the paper is unusually honest — full prompts, ablation details, and explicit admissions that GSS and GPT-4o-mini gains are weak. The evaluation is also external: labels come from six non-WVS surveys, so the results don't reduce to fitted values. That is real credit.\n\nThe soft spots, in order of severity. First, the test set. Appendix B.2 says the 2,000 evaluation points are k-means centroid prototypes — 441 from AFRO, 74 from CGSS, etc. A centroid in embedding space has no survey response attached to it. The paper never says whether each centroid is mapped to the nearest real respondent (and how label conflicts are resolved) or whether accuracy is computed against some synthetic label. If it's the latter, Table 1 is not measuring human responses at all. That is the first thing I'd ask the authors to clarify, and it is fixable: spell out the mapping or re-run on actually sampled respondents.\n\nSecond, hyperparameters. K=5 and N=3 are selected by test-set accuracy (Section 5.1.1 and H.3), and Table 1 reports those tuned settings while baselines use fixed, untuned configurations. On GPT-4o-mini, where the gain over ValuesRAG is +0.5 points, the tuning effect visible in Figure 3 (0.02–0.05 swings for K) is large enough to swallow the headline claim. That's a fair criticism, not a manufactured one. Third, smaller issues: the Self-consistency baseline appears to run at temperature 0, which removes the sampling diversity it needs, and there is no N=0 condition, so the ontology's marginal contribution is never isolated.\n\nThe demographic-transfer assumption the reader flagged is real but not the main threat. The pipeline's retrieval channel could be wrong and the system could still win on these benchmarks; the centroid protocol determines whether these benchmarks exist at all.\n\nWho should read this: anyone working on cultural alignment or ontology-grounded RAG will find the design and the prompts useful, but the empirical claims should be treated as provisional until the evaluation protocol is clarified. It deserves a serious referee — the system is novel and the flaws are disclosed and fixable, not concealed — but I'd expect major revision before acceptance.\n\nRecommendation: send to peer review with instructions to the authors to specify exactly what the 2,000 test instances are and how gold labels are attached; if centroids were used directly, re-run on real respondents.","headline":"Coherent, unusually transparent system paper whose evaluation protocol has a potentially load-bearing hole: the test 'instances' appear to be k-means centroids, not labeled respondents.","tokens_in":33203,"tokens_out":1574,"would_cite":false,"duration_ms":20409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OG-MAR claims that structuring value knowledge as an ontology and running demographically matched personas makes LLMs answer value surveys more accurately.","keywords":["cultural alignment","ontology-guided reasoning","multi-agent LLM","World Values Survey","value personas","retrieval-augmented generation","demographic grounding","cultural bias mitigation"],"falsifier":"Take a set of target respondents and their true answers; run OG-MAR with the ontology triples replaced by random triples drawn from the same ontology, and again with no triples. If accuracy does not fall substantially, the ontology is not carrying the claimed weight. A second check: use target respondents whose demographics match WVS respondents but whose values are known to diverge (e.g., immigrant versus native-born within the same demographic cell); if OG-MAR still predicts from demographic match, the demographic-proximity channel is over-trusting resemblance.","tokens_in":32003,"feed_emoji":"🌍","tokens_out":3988,"duration_ms":43613,"temperature":0.7,"pith_summary":"OG-MAR argues that LLMs align better with regional values when the missing structure is supplied: raw survey evidence is organized into a 76-class value ontology with relational triples, and inference runs through several value-persona agents grounded in demographically similar World Values Survey respondents, with a judgment agent synthesizing their outputs. The paper tests this on six regional social surveys—Europe, the US, China, India, Africa, and Latin America—across four LLM backbones, reporting higher average accuracy than zero-shot prompting, role assignment, self-consistency, debate, and retrieval-augmented baselines. The authors also report that the resulting reasoning traces are more grounded and consistent in human evaluation. If correct, the result would mean cultural alignment can be improved without retraining, by giving the model structured, empirically grounded value relations plus demographic evidence.","feed_headline":"Ontology-guided agents lift LLM cultural alignment scores","feed_subtitle":"Structured value relations plus demographically matched personas beat prompting and retrieval baselines on six regional surveys.","key_machinery":"The central machinery is the cultural value ontology plus the persona-judge loop. The ontology is a fixed taxonomy of 12 value domains and 76 fine-grained value categories connected by 150 directed, human-reviewed relational triples (e.g., \"Religious Exclusivism severely undermines Outgroup Tolerance\"), generated from competency questions and region-conditioned LLM reasoning. At inference, dense retrieval selects demographically similar WVS respondents whose value summaries become individual personas; each persona reasons under the retrieved ontology triples, and a judgment agent combines their outputs by evidence strength, vote only in near-ties, and demographic relevance. The ontology does","core_discovery":"The central claim is that ontology-guided multi-agent reasoning improves cultural alignment over competitive baselines. The pipeline first converts every WVS respondent into topic-wise value summaries under a fixed taxonomy, then builds a global cultural ontology of 150 directed relations between 76 value categories by asking LLMs competency questions conditioned on regional respondent data and consolidating with human experts. At inference, a question is routed to relevant topics, ontology triples and demographically similar WVS respondents are retrieved, and several value-persona agents answer from their own simulated worldview; a judgment agent then applies an evidence-first protocol rath","pith_inferences":["An implication the authors leave implicit: the demographic-to-value transfer could be tested by swapping the retrieval corpus (e.g., using each regional survey's own respondents as evidence) and checking whether accuracy rises or falls; if the WVS corpus is replaceable, the method's contribution is the ontology-plus-persona scaffold, not the WVS grounding.","The ontology is built once globally; one could infer that region-specific or question-specific ontologies might outperform it, since cross-cultural value relations are not necessarily stable across all 64 WVS countries.","A testable extension: replace ontology triples with corpus-derived co-occurrence statistics to see whether the human-reviewed relations add signal beyond statistical association.","Because the judgment agent never sees the ontology directly, it relies on persona traces to carry relational structure; an editorial guess is that injecting the ontology at the judgment stage too might strengthen consistency in ambiguous cases."],"forward_implications":["Cultural alignment can be improved at inference time by supplying structured value relations and demographic evidence, without fine-tuning the underlying model.","On the paper's numbers, ontology-guided retrieval beats retrieving raw value snippets, suggesting that relational structure, not just evidence, carries the gain.","The largest accuracy gains appear on CGSS (China) and ISD (India), contexts farthest from dominant pretraining priors, implying the method helps most where culture-default bias is strongest.","The reasoning traces produced by personas and the judgment agent can be inspected as evidence-first chains, which the human evaluation rates as grounded and consistent.","The framework transfers to new target surveys by re-running the retrieval against the same WVS corpus, as demonstrated on EVS, GSS, CGSS, ISD, LAPOP, and Afrobarometer."],"fun_headline_variants":["Ontology-guided multi-agent reasoning boosts LLM cultural fit","Multi-agent ontology method aligns LLMs with cultural values","Cultural alignment via ontology and persona agents","OG-MAR: ontology agents improve LLM cultural alignment"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that demographically similar people in the World Values Survey answer value questions the way the target respondent from a different survey would, and that the topic classifier identifies the right value categories for each question.","fun_headline_variants_meta":{"raw":{"variants":["Ontology-guided multi-agent reasoning boosts LLM cultural fit","Multi-agent ontology method aligns LLMs with cultural values","Cultural alignment via ontology and persona agents","OG-MAR: ontology agents improve LLM cultural alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000118,"raw_usage":{"total_tokens":877,"prompt_tokens":663,"completion_tokens":214,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":152}},"tokens_in":407,"tokens_out":214,"duration_ms":2779,"temperature":1.0,"reasoning_tokens":152,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:54:03.561910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of target respondents and their true answers; run OG-MAR with the ontology triples replaced by random triples drawn from the same ontology, and again with no triples. If accuracy does not fall substantially, the ontology is not carrying the claimed weight. A second check: use target respondents whose demographics match WVS respondents but whose values are known to diverge (e.g., immigrant versus native-born within the same demographic cell); if OG-MAR still predicts from demographic match, the demographic-proximity channel is over-trusting resemblance.","supporting_citations":[],"review_version":1}