REVIEW 3 major objections 4 minor 26 references
OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read OmniSapiens-7B 2.0, trained with HARPO—a reinforcement-learning method that modulates policy-update advantages to balance heterogeneous tasks and samples—achieves the strongest multitask and held-out performance among compared behavioral mo
desk verdict HARPO's advantage modulation is a sensible idea, but the comparison against GRPO is confounded because HARPO drops the KL term, and the paper needs a no-KL GRPO control, seeds, and variance before the central claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the HARPO-modulated advantage, formed by multiplying the group-normalized advantage of each sampled response by a sample-level and a task-level modulation factor. Each factor is the reciprocal of a contribution signal (average absolute group-normalized advantage of a sample or task) divided by a geometric-mean reference; this construction guarantees the factors have geometric mean 1, so the overall scale of the policy update is unchanged. The ratio-based design tempers contribution signals that can vary by orders of magnitude, and the inertial control—exponential moving average on signals, multiplicative updates on factors—lets the reweighting evolve more slowly than th
What would settle it
Train HARPO and a control version that uses the same smoothing and geometric centering but with randomly assigned per-sample and per-task modulation weights; if the control matches HARPO's average rank across tasks, the contribution-signal mechanism is not the driver. A more direct check: during training, measure the correlation between the contribution signal and the actual per-sample gradient norm; a near-zero correlation would falsify the proxy on which the method is built.
Extended reading notes
Core claim
The central claim is that HARPO rebalances the gradient contributions of heterogeneous behavioral tasks by modulating group-normalized reward advantages before the policy update. For every sample and every task, the method computes a contribution signal—the average absolute group-normalized advantage across that sample's or task's rollouts—and rescales advantages by the reciprocal of the signal's ratio to a geometric-mean reference (one reference within each task, one across tasks). Because the modulation factors are constructed as reciprocals of ratios to their geometric mean, they multiply to 1, so upscaling some advantages is exactly offset by downscaling others and the global update scal
Load-bearing premise
The load-bearing premise is that a sample's or task's average absolute group-normalized advantage faithfully reflects how much it contributes to the policy update; the true contribution also depends on the size of the log-probability gradient, which can vary across tokens and samples.
Editorial extensions
If this is right
- Unified behavior models can be trained across heterogeneous tasks without hand-crafted per-task weights or a separate value network.
- Weak tasks that collapse under the base group-based method—sarcasm and humor detection in the reported runs—keep improving under HARPO (sarcasm F1 rises from about 54% to about 71%).
- The rebalanced model transfers zero-shot to held-out behavior domains such as autism behavior recognition and speech emotion recognition, outperforming previous unified behavior models.
- Because the method only reuses rewards already present in policy-gradient training, it can be added to existing on-policy optimization loops with two extra smoothing hyperparameters.
Reading between the lines
- The same two-level reweighting recipe could apply to any multitask reinforcement-learning setting with heterogeneous reward scales—for example, combining coding, math, and dialogue rewards—though the paper demonstrates it only on social-behavior tasks.
- A natural stress test: apply HARPO to a synthetic benchmark where one task's reward variance is artificially inflated while its informativeness is held fixed; if the model still avoids collapse, the contribution-signal proxy is doing real work rather than the smoothing.
- If balanced training indeed yields more transferable features, HARPO-trained models may serve as strong initializations for data-scarce clinical and social domains; the zero-shot results reported here are suggestive but would need broader validation.
- The contribution proxy could be replaced by actual per-token gradient-norm estimates; comparing the two would reveal whether the proxy or the modulation structure is the bottleneck.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HARPO, an extension of GRPO for multitask behavioral training. HARPO computes sample- and task-level contribution signals as average absolute group-normalized advantages, converts them into inverse-geometric-mean modulation factors, and applies inertial smoothing before the PPO-style clipped update. The authors train Qwen2.5-Omni-7B on the 10-task Human Behavior Atlas benchmark, reporting that the resulting model, OmniSapiens-7B 2.0, achieves the best average rank across existing models and held-out AUT/SER transfer, and that HARPO outperforms GRPO, RLOO, RE++, and GPG with the same reward design and base model. The paper also includes ablations of the modulation components and qualitative reasoning-trace analyses.
Significance. If the attribution is supported, HARPO is a useful and generally applicable contribution to critic-free multitask reasoning RL: it gives a concrete recipe for reweighting advantages at two levels, provides an algorithm block (Algorithm 1), includes ablations of each design choice, and evaluates on an external held-out task (AUT, SER) in addition to the in-distribution benchmark. The paper is also transparent about experimental details and plans to release code and reasoning traces. However, the central causal claim — that the advantage-modulation mechanism, rather than the accompanying removal of KL regularization or other hyperparameter changes, drives the gains — is not currently established by the reported experiments. The main comparison against GRPO is confounded, and the absence of multiple seeds leaves rank differences of the size used to support the ablations within the range of training noise.
major comments (3)
- [§3.2, Eq. (12); App. C.1; Table 1] The headline HARPO-vs-GRPO comparison is confounded by KL regularization. Eq. (12) displays a KL penalty term, but App. C.1 states that HARPO 'omit[s] explicit KL regularization' because its modulation acts as regularization, while all RL baselines 'reuse the hyperparameter configurations reported in the original papers' — which, for GRPO, includes the KL penalty. Thus the comparison changes two variables at once, and the reported gains (e.g., +42.29% on SAR in Table 1) could arise from removing the KL term alone. The ablations in Table 3 are all within HARPO's no-KL setting, so they cannot isolate the modulation mechanism. The authors should add a GRPO run with KL removed under the same reward and early-stopping settings, and ideally a HARPO run with KL enabled, before claiming that heterogeneity-aware advantage modulation is responsible for the improvement. They should also reconcile E
- [§4.2–§4.5, Tables 1–3] No error bars, multiple seeds, or significance tests are reported for any training run. On-policy RL with only 5 rollouts per sample is high-variance, and the rank differences used to support the method and its ablations are small in several cases (e.g., HARPO 1.90 vs. w/o structured modulation 2.00 in Table 3; HARPO 1.70 vs. RLOO/GPG 2.50 in Table 1). A single seed is insufficient to distinguish the proposed mechanism from noise. The authors should report mean ± std over at least three seeds for HARPO, for GRPO, and for the ablations, and should state whether differences are statistically reliable. Additionally, the hyperparameter policy is asymmetric: baselines use their original-paper hyperparameters while HARPO uses beta_rho = beta_s = 0.95 and no KL; a fair comparison should also tune or at least vary the baselines' KL weights.
- [§3.2, Eqs. (6)–(7) and Eq. (5); §4.5] The load-bearing proxy assumption is not directly validated. The paper equates a sample's or task's contribution to the policy update with the average absolute group-normalized advantage (Eqs. (6)–(7)). The actual per-rollout gradient is A * grad log pi (Eq. (5)), whose norm varies across tokens and sequence lengths, so the average |A| may be a poor ranking of true gradient contributions. Figure 4 shows a correlation between relative task advantage and reward for one task, but it does not test whether the proxy ranks samples correctly. I request a diagnostic: on a small training subset, compute the actual gradient-norm contribution per sample/task and compare it with p(t), or replace p(t) with a fixed/random/ordinal signal in the algorithm. Without such evidence, the reported gains could come from the geometric centering, the inertial smoothing, or the no-KL setting rather than from vali
minor comments (4)
- [Abstract vs. §1] The abstract reports gains of up to +12.02% and +9.37%, while the introduction reports +16.85% and +9.37% for multitask and held-out settings. Please clarify which number corresponds to which comparison and make the abstract consistent.
- [Throughout] There are several typographical issues: 'balances leaning' in §1, 'simulateneously' in App. D.2, and 'of of training' in §4.5. Please proofread.
- [Table 1] Several numeric entries run together, e.g., '98.3990.68' and '64.7030.40'. The table also lists HARPO under both Models and Training Algorithms, which is understandable but should be labelled clearly to avoid the appearance of duplicated rows.
- [Figure/Section cross-references] In §4.5, 'App. E.1 Fig. 5' should presumably refer to the main-text Fig. 5 or to the appropriate appendix figure; several appendix references are ambiguous because the main text already has a Fig. 5.
Circularity Check
No load-bearing circularity: HARPO's advantage-modulation mechanism is defined independently of the reported outcomes, and the central results are checked against external held-out benchmarks; the main concerns (self-authored benchmark, GRPO KL-regularization asymmetry) are experimental-control issues, not definitional circularity.
full rationale
The derivation chain is self-contained. HARPO constructs contribution signals p from average absolute group-normalized advantages (Eqs. 6-7), defines geometric-mean-centered modulators (Eqs. 8-10), and substitutes the modulated advantage into the GRPO-style objective (Eq. 12). Nothing in this chain is fitted to or defined in terms of the reported evaluation outcomes: the modulation factors depend only on training-batch advantages, and performance is measured on an independent benchmark. The held-out AUT (AV-ASD) and SER (IEMOCAP) evaluations are external datasets, and comparisons against external baselines (Gemma, Qwen, HumanOmniV2) do not reduce to the paper's own constructions. The self-citation of the Human Behavior Atlas (Ong et al., 2026) is a legitimate concern about benchmark independence, but the benchmark is not invoked as a theorem or as a substitute for the derivation; the paper also validates on external data, so the central claim has independent content. The Appendix C.1 asymmetry in which HARPO omits KL regularization while GRPO retains it is a real control/attribution problem for the +42.29% claim, but it is a confound, not circularity: the HARPO vs GRPO comparison could be explained by the missing KL penalty, yet that does not show that HARPO's results reduce to its inputs by construction. Similarly, the claim that average absolute advantage is a proxy for gradient contribution (Eqs. 4-7) is an assumption, not a definitional identity with the prediction. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is smuggled in via self-citation.
Assumptions & free parameters
free parameters (5)
- beta_rho (contribution signal EMA decay) =
0.95
- beta_s (modulation factor smoothing) =
0.95
- w_fmt (format reward weight) =
0.2
- lambda_len (length penalty scale) =
0.75
- L_max and B (length penalty thresholds) =
812 and 128 tokens
assumptions (4)
- standard math Policy Gradient Theorem
- domain assumption Average absolute advantage is a valid proxy for gradient contribution
- domain assumption LLM judge provides valid labels for free-text QA tasks
- domain assumption Human Behavior Atlas is a fair and representative benchmark
Cite this review
Pith. "Pith review of OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization." pith.science (2026). https://pith.science/paper/KEIFX3TH
@misc{pith2026260210635,
author = {Pith},
title = {Pith review of: OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/KEIFX3TH}},
note = {Machine review of arXiv:2602.10635}
}
read the original abstract
Socially intelligent AI systems must reason across diverse human behavioral tasks and generalize to new social contexts. However, behavioral data is inherently heterogeneous, comprising diverse modalities and prediction targets that produce uneven training signals across samples, creating imbalanced learning dynamics that challenge existing AI models. To address this, we develop Omnisapiens-7B 2.0, a foundation model for social behavior processing that explicitly addresses learning from heterogeneous behavioral data. This is enabled through Heterogeneity-Aware Relative Policy Optimization, a new RL method that rebalances learning signals across samples by approximating each sample's contribution to the policy update and using these estimates to drive geometrically centered, inertially smoothed advantage modulation for stable training. Omnisapiens-7B 2.0 achieves the best and most consistent performance across 10 behavioral tasks, while also attaining the best performance on all five held-out benchmarks, with gains of up to +12.02% and +9.37% respectively. Furthermore, it demonstrates more consistent and interpretable reasoning traces, supporting reliable real-world behavioral applications. Our model is available at https://github.com/MIT-MI/human_behavior_atlas.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Ahmadian, A., Cremer, C., Gall´e, M., Fadaee, M., Kreutzer, J., Pietquin, O., ¨Ust¨un, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,
-
[4]
Constrained dynamical neural ode for time se- ries modelling: A case study on continuous emotion prediction
Dang, T., Dimitriadis, A., Wu, J., Sethu, V ., and Ambikaira- jah, E. Constrained dynamical neural ode for time se- ries modelling: A case study on continuous emotion prediction. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,
2023
-
[5]
To enable fair 4https://huggingface.co 5https://github.com/volcengine/verl 14 OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization comparison with HARPO, all reinforcement learning methods are run on the same Human Behavior Atlas benchmark, with the same base model, Qwen 2.5-Omni-7B (Xu et al....
2025
-
[6]
Ablation results for HARPO components, grouped by behavioral tasks and datasets. Following the unified metrics proposed in the Human Behavior Atlas Benchmark (Ong et al., 2026), we use binary weighted F1 for SEN; mean per-class weighted accuracy for EMO; weighted F1 for HUM, SAR, ANX, DEP, PTSD; and LLM-Judge accuracy for SOC, INT, NVC. Model / Ablations ...
2026
-
[7]
A closer look at deep policy gradients.arXiv preprint arXiv:1811.02553,
Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. A closer look at deep policy gradients.arXiv preprint arXiv:1811.02553,
- [8]
-
[13]
Avec 2016: Depression, mood, and emotion recognition workshop and challenge
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., and Pantic, M. Avec 2016: Depression, mood, and emotion recognition workshop and challenge. InProceed- ings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 3–10. ACM,
2016
-
[14]
A novel markovian framework for integrating absolute and rela- tive ordinal emotion information.IEEE Transactions on Affective Computing, 14(3):2089–2101,
Wu, J., Dang, T., Sethu, V ., and Ambikairajah, E. A novel markovian framework for integrating absolute and rela- tive ordinal emotion information.IEEE Transactions on Affective Computing, 14(3):2089–2101,
Show all 26 references
-
[16]
Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
-
[18]
and Zuo, C
Zhang, J. and Zuo, C. Grpo-lead: A difficulty-aware re- inforcement learning approach for concise mathemat- ical reasoning in language models.arXiv preprint arXiv:2504.09696,
-
[19]
The weighted F1 is then computed as: Weighted-F1= X c∈C nc N ·F1 c, wheren c is the number of true instances in classc,Nis the total number of instances, andCis the set of classes
For HUM (Humour Detection), SAR (Sarcasm Detection), DEP (Depression Detection), ANX (Anxiety Detection), and PTSD (PTSD Detection), the weighted F1 score is computed: F1 = 2·Precision·Recall Precision+Recall , where, Precision= T P T P+F P,Recall= T P T P+F N. The weighted F1...
2018
-
[20]
On the other hand, we implement the reinforcement learning training algorithms in Tab
are taken from the Human Behavior Atlas benchmark paper (Ong et al., 2026)), while we run the evaluation of Qwen 3-VL-8B Instruct (Xu et al., 2025b) using its implementation on Huggingface4. On the other hand, we implement the reinforcement learning training algorithms in Tab....
2026
-
[22]
Finally, we provide a overlong length penalty, rlen which follows Zhang & Zuo (2025) to prevent excessive length and verbosity of responses
This check is applied to the full response and enforces adherence to the prescribed output schema. Finally, we provide a overlong length penalty, rlen which follows Zhang & Zuo (2025) to prevent excessive length and verbosity of responses. Accordingly, let ℓ be response length...
2025
-
[23]
Each value is the arithmetic mean over datasets associated with the task
Per-task performance (%) for HARPO ablations across behavioral tasks. Each value is the arithmetic mean over datasets associated with the task. Avg. Rank is computed across tasks using per-task ranks (higher is better; ties use average rank), and then averaged over tasks (lowe...
-
[25]
Additional Training Plots We provide additional training plots to empirically illustrate the training dynamics in our experiments
E.3. Additional Training Plots We provide additional training plots to empirically illustrate the training dynamics in our experiments. We include a Fig. 7 that depicts the advantage distributions of the different behavioral tasks in the Human Behavior Atlas benchmark (Ong et ...
2026
-
[26]
Full results grouped by behavioral tasks (headers) and their relevant datasets (sub-headers). Following the unified metrics proposed in the Human Behavior Atlas Benchmark (Ong et al., 2026), we use binary weighted F1 for SEN; mean per-class weighted accuracy for EMO; weighted ...
2026
-
[1999]
Gemma 3 technical report.arXiv preprint arXiv:2503.19786,
Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,
-
[2014]
R., Solar-Lezama, A., and Liang, P
Li, H., Tjandrasuwita, M., Fung, Y . R., Solar-Lezama, A., and Liang, P. P. Mimeqa: Towards socially- intelligent nonverbal foundation models.arXiv preprint arXiv:2502.16671,
-
[2015]
Proximal policy optimization algorithms
10 OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2017]
Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2018]
K., Rahman, W., Zadeh, A
Hasan, M. K., Rahman, W., Zadeh, A. B., and Zhong, J. Ur- funny: A multimodal language dataset for understanding humor. InEMNLP-IJCNLP, pp. 2046–2056,
-
[2020]
MOSI: Multimodal corpus of sentiment intensity and subjectiv- ity analysis in online opinion videos.arXiv preprint arXiv:1606.06259,
Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P. MOSI: Multimodal corpus of sentiment intensity and subjectiv- ity analysis in online opinion videos.arXiv preprint arXiv:1606.06259,
-
[2022]
Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y ., Dang, K., et al. Qwen2. 5-omni techni- cal report.arXiv preprint arXiv:2503.20215, 2025a. Xu, J., Guo, Z., Hu, H., Chu, Y ., Wang, X., He, J., Wang, Y ., Shi, X., He, T., Zhu, X., et al. Qwen3-omni...
-
[2023]
Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,
Hu, J. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,
-
[2024]
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,
-
[2025]
Gpg: A simple and strong reinforcement learning baseline for model reasoning.arXiv preprint arXiv:2504.02546,
Chu, X., Huang, H., Zhang, X., Wei, F., and Wang, Y . Gpg: A simple and strong reinforcement learning baseline for model reasoning.arXiv preprint arXiv:2504.02546,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.