Pith. sign in

REVIEW 4 major objections 6 minor 34 references

FairReason: Balancing Reasoning and Social Bias in MLLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper attempts to establish that a roughly 1:4 mix of debias-focused to reasoning-centric training data, trained with rule-based reinforcement learning, cuts stereotype scores by about 10% while retaining 88% of original reasoning…

desk verdict The systematic data-mix sweep is worth reading, but the headline 10%/88% claim doesn't match their own tables and a figure contradicts the table by 2x. read the letter →

arxiv 2507.23067 v2 pith:377ERFF6 submitted 2025-07-30 cs.AI

Zhenyu Pan , Yutong Zhang , Jianshu Zhang , Haoran Lu , Haozheng Luo , Yuwei Han , Philip S. Yu , Manling Li
show 1 more author
Han Liu
This is my paper
classification cs.AI
keywords multimodallargelanguagemodelssocialbiasmitigationreasoningtrade-offreinforcementlearningGRPOdatamixturestereotypepost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reasoning gains and social-bias mitigation inherently trade off in multimodal large language models, and whether a practical balance exists under a fixed training budget. It benchmarks supervised fine-tuning, knowledge distillation, and rule-based reinforcement learning on identical debiasing data, then sweeps the proportion of debias-focused versus reasoning-focused training samples within each paradigm. The central finding is a consistent sweet spot: a roughly 1:4 mix trained with reinforcement learning cuts stereotype scores by about 10% while retaining 88% of the original reasoning accuracy. If this regularity holds beyond the tested models and benchmarks, it gives developers a concrete, budget-limited recipe for reducing social bias without sacrificing much reasoning ability.

What carries the argument

The central mechanism is the systematic sweep over the debias-to-reasoning data ratio under a fixed budget, combined with rule-based reinforcement learning (GRPO) as the training paradigm. GRPO is a group-relative policy optimization method that updates the model by comparing groups of sampled outputs; the sweep varies the proportion of bias-centric samples (0%, 5%, 10%, 20%, 40%, 100%) to chart the trade-off curve and locate the operating point where bias reduction is large and reasoning loss is small.

What would settle it

Evaluate models trained with the 1:4 GRPO recipe on a held-out, disjoint sample of BBQ and VLBiasBench plus an independent bias benchmark; if the stereotype-score reduction drops to near zero or fails to reach about 10%, the sweet-spot claim is an artifact of train/eval overlap.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an empirical regularity: across three model families (Qwen3-8B, Qwen2.5-VL-7B, InternVL3-8B) and three training paradigms (SFT, distillation, and GRPO), the relationship between reasoning accuracy and bias reduction follows a consistent trade-off curve, and the best operating point sits at roughly 20% debias-centric data, a 1:4 mix, trained with GRPO. At that point, stereotype scores drop about 10% while reasoning accuracy retains about 88% of the untrained model's level. The paper also reports that GRPO outperforms SFT and distillation for bias mitigation across all tested models and scenarios, and that beyond 20% bias-centric data, bias gains shrink while reasoning declines more steeply.

Load-bearing premise

The measured bias reduction is genuine debiasing rather than fitting to the BBQ and VLBiasBench distributions, since training and evaluation samples come from the same benchmark pools and no disjoint split or independent bias benchmark is described.

Editorial extensions

If this is right

  • Across the three tested model families, GRPO consistently beats supervised fine-tuning and knowledge distillation at reducing social bias while preserving reasoning.
  • A 20% debias / 80% reasoning data split is the best operating point; moving from 20% to 100% bias data adds only a few points of bias accuracy while accelerating reasoning decline.
  • The 1:4 mix under RL delivers about a 10% stereotype-score reduction and retains about 88% of original reasoning accuracy.
  • Reasoning improvements do not automatically make outputs fairer, so fairness must be treated as a separate objective during post-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sweet spot is driven by benchmark distribution rather than general debiasing, the 1:4 recipe may not transfer to an independent bias benchmark; a held-out evaluation would settle that.
  • The 88% reasoning-retention figure is measured on math and geometry benchmarks; a broader reasoning suite could shift the preferred ratio.
  • The paper's limited training budget (about 3,000 debias samples, one epoch, LoRA) suggests the sweet spot may move with scale, so larger models or longer training could require a different mix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical study of how reasoning-focused post-training interacts with social-bias mitigation in small LLMs and MLLMs (Qwen3-8B, Qwen2.5-VL-7B, InternVL3-8B). It compares three training paradigms—supervised fine-tuning, knowledge distillation, and rule-based reinforcement learning (GRPO)—under shared settings, then sweeps the proportion of debias-oriented versus reasoning-oriented training samples. The central claim, stated in the abstract and conclusion, is that a roughly 1:4 debiasing-to-reasoning mix trained with GRPO reduces stereotype scores by about 10% while retaining about 88% of original reasoning accuracy, and that this 'sweet spot' is consistent across models and strategies. Bias is measured on BBQ and VLBiasBench; reasoning is measured on AIME 2024, MATH-500, MathVerse, and Geometry3K. The paper also releases models on Hugging Face.

Significance. If the central empirical claim were fully supported, the paper would provide useful, actionable guidance for practitioners training small MLLMs under limited budgets, and it would be a rare systematic comparison of three training paradigms for bias mitigation under matched conditions. The study design has real strengths: it covers multiple model families, uses both text-only and multimodal benchmarks, and includes data-mix sweeps rather than a single configuration. The release of trained models on Hugging Face is a positive step for reproducibility. However, the headline quantitative result is not verifiable from the data reported in the paper, and the evaluation protocol appears to allow overlap between training and evaluation samples from the same benchmark family. As a result, the current manuscript does not establish the claimed trade-off curve or the 10%/88% sweet spot.

major comments (4)
  1. [Abstract / Section 5 (Conclusion) vs. Table 4] The headline claim that a 1:4 mix under GRPO 'cuts stereotype scores by 10% while retaining 88% of the model's original reasoning accuracy' is not supported by the paper's own results table. From Table 4, the GRPO 20% debias column for Qwen3-8B on BBQ moves accuracy from 0.82 to 0.87, a relative increase of 6.1% (or a relative error-rate decrease of 27.8%); for Qwen2.5-VL-7B on VLBiasBench it moves from 0.78 to 0.82, a relative increase of 5.1% (or a relative error-rate decrease of 18.2%). Neither equals 10%. No definition of 'stereotype score' is given anywhere in the paper, and no value of 10% appears in Tables 3 or 4. The authors must define the stereotype-score metric precisely and report it for every configuration, or revise the abstract and conclusion to match the numbers actually measured.
  2. [Section 4.1 and Section 3.4] The bias-mitigation evaluation appears to be conducted on subsets of the same data used for training. Section 4.1 states that the authors 'evaluate their performance on different subsets of the training data'; Section 3.4 says a fixed sample of 3,000 entries from BBQ and VLBiasBench is used for training; Section 3.5/4.1 then evaluates on 5,000-sample subsets of the same benchmarks. The paper never states that the evaluation subsets are disjoint from the training samples, and no independent or unseen bias benchmark is used. If the evaluation sets overlap the training sets, the reported bias reductions could reflect memorization of benchmark distributions rather than genuine debiasing. The authors must either provide an explicit disjoint-split protocol (e.g., by item ID) with verification, or evaluate on a held-out bias benchmark not seen during training.
  3. [Figure 2 vs. Table 4] There is a large unexplained discrepancy between Figure 2 and Table 4 for the pre-GRPO VLBiasBench baseline. Figure 2's caption reports Qwen2.5-VL and InternVL3 rising from 0.35/0.38 to 0.54 after GRPO, whereas Table 4 reports the same Qwen2.5-VL-7B model at a 0.78 baseline and 0.82 at 20% debias data. If the figure reports a different metric (e.g., ambiguous-scene accuracy only) or a different evaluation subset, the caption and table must say so explicitly. As presented, the factor-of-two disagreement about the baseline makes it impossible to assess the claimed effect size or the 'consistent sweet spot' claim.
  4. [Section 8 and Tables 3–4] All results are single-seed (seed 42), and although the sampling strategy generates 5 responses per prompt, the tables report no variance, confidence intervals, or significance tests. The paper repeatedly uses the word 'consistent' to describe the sweet spot across models and training schemes, but with one run per condition there is no statistical basis for that claim. At minimum, the authors should report standard deviations or bootstrap intervals over seeds and over the 5 sampled responses, and ideally a significance test for the 20%-vs-0% comparisons.
minor comments (6)
  1. [Section 3.4] The text says the sweep varies 'proportions of reasoning data' (5%, 10%, 20%, 40%), while Section 4.2 and Figure 3 describe the x-axis as the proportion of bias-centric data. Please clarify which quantity the columns in Tables 3 and 4 represent.
  2. [Section 3.1, Research Question 1] The phrase 'generational social bias' appears to be a typo; presumably 'general social bias' or a specific category such as 'gender' was intended.
  3. [Section 8] The sampling strategy says 'maximum response token limit of 10,000,' which is unusually large for evaluation responses; please confirm whether this is a typo and state the actual decoding settings used for evaluation.
  4. [Section 4.2] The sentence 'we normalize the test results and plot them' does not specify the normalization procedure; please state exactly how scores were normalized before plotting.
  5. [Tables 3 and 4] The tables list no sample sizes for the evaluation subsets and no error bars; adding N per benchmark-cell and the number of evaluation prompts would improve reproducibility.
  6. [Appendix] Please provide the evaluation prompts and the exact sampling code or a data-version identifier so that the disjointness of train/eval subsets can be independently checked; model release alone does not enable verification of the bias measurements.

Circularity Check

1 steps flagged · score 6.0 of 10

Bias-mitigation 'sweet spot' is evaluated on the same BBQ/VLBiasBench data from which the 3k debias training traces were sampled, with no stated disjointness; the 10% stereotype reduction is therefore partly an in-distribution fit rather than an independent prediction. The reasoning axis uses independent benchmarks, but the bias axis is central.

  1. fitted input called prediction [Section 3.3–3.4 (training data) and Section 4.1 (evaluation)]
    "We utilize a fixed sample of 3k entries from the BBQ and VLBiasBench datasets for model training... We employ three training strategies to train three different models and evaluate their performance on different subsets of the training data."

    The debias training traces are sampled from BBQ and VLBiasBench, and the bias evaluation is described as subsets of the same 'training data,' with no statement that the 5k evaluation subsets exclude the 3k training samples. The abstract's headline result—'cuts stereotype scores by 10%'—is thus measured on the same benchmark distribution that supplied the debiasing supervision signal. The measured bias improvement is at least partly a fit to the benchmark rather than an independent, held-out estimate of general debiasing. The reasoning-retention numbers come from independent reasoning benchmarks, so only the bias axis is affected, but that axis is central to the paper's practical recommendation.

full rationale

FairReason is an empirical benchmarking paper, not a formal derivation; most of its claimed contributions are descriptive comparisons of trained models. Self-citations in the related-work and reference sections are routine and are not load-bearing for the central claim, so they do not contribute to circularity. The only circularity-adjacent defect is the train/evaluation design on the bias axis: Section 3.3 samples 3k reasoning traces from BBQ and VLBiasBench for training, and Section 4.1 evaluates on 5k subsets of the same benchmarks, explicitly calling the evaluation subsets 'subsets of the training data' and never stating disjointness. If the evaluation subsets overlap with the training traces, the reported 10% stereotype reduction is forced by in-distribution fitting rather than independently predicted. Even under the authors' own Table 4, the headline number is not reproducible: for GRPO, Qwen3-8B BBQ accuracy moves from 0.82 to 0.87 (a 6.1% relative gain; a 27.8% relative error-rate reduction), and Qwen2.5-VL-7B VLBiasBench moves from 0.78 to 0.82 (a 5.1% relative gain; an 18.2% error-rate reduction); neither equals 10%. Figure 2 reports pre-GRPO VLBiasBench scores of 0.35/0.38 rising to 0.54, while Table 4 reports the same model at 0.78 baseline and 0.82 at 20%, a factor-of-two disagreement that prevents verification of the central claim. These latter problems are internal-consistency and verifiability issues rather than circularity per se, which is why the score is 6 rather than higher: the bias claim is partially circular by construction, while the reasoning-retention claim has independent support and the paper's overall framework is not a self-referential derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of BBQ/VLBiasBench as bias measures and on the assumption that training on these benchmarks generalizes. The trade-off claim also depends on the chosen 20% ratio from a coarse sweep, plus the implicit assumption that a one-epoch, 5k-sample budget captures the phenomenon.

free parameters (2)
  • Debias data proportion (sweet spot) = 20% (1:4 debias-to-reasoning mix)
    Selected as best trade-off from a sweep over 5%, 10%, 20%, 40%, and 100% debias data; the headline result depends on this being the right ratio.
  • Training budget = 3k training traces, about 5k total samples
    Chosen to keep the study small; the trade-off may shift with larger budgets, which the paper acknowledges.
assumptions (4)
  • domain assumption BBQ and VLBiasBench are valid measures of social bias in LLMs and MLLMs.
    Bias mitigation is quantified solely through these two benchmarks; no independent validation is provided.
  • domain assumption Training on 3k BBQ/VLBiasBench samples and evaluating on 5k subsets of the same benchmarks measures generalizable bias mitigation.
    Unless the split is disjoint and representative, the measured improvement may reflect memorized benchmark patterns rather than general debiasing.
  • domain assumption A single epoch and small sample budget are sufficient to identify stable trade-offs.
    All sweeps use one epoch and a fixed seed 42; no multi-seed or multi-budget verification is reported.
  • domain assumption MATH-500, AIME 2024, MathVerse, and Geometry-3K accurately represent reasoning ability.
    Reasoning retention is measured on these four math-centric benchmarks, which may not cover general reasoning capabilities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FairReason: Balancing Reasoning and Social Bias in MLLMs." pith.science (2026). https://pith.science/paper/377ERFF6

@misc{pith2026250723067,
  author       = {Pith},
  title        = {Pith review of: FairReason: Balancing Reasoning and Social Bias in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/377ERFF6}},
  note         = {Machine review of arXiv:2507.23067}
}
read the original abstract

Multimodal Large Language Models (MLLMs) already achieve state-of-the-art results across a wide range of tasks and modalities. To push their reasoning ability further, recent studies explore advanced prompting schemes and post-training fine-tuning. Although these techniques improve logical accuracy, they frequently leave the models' outputs burdened with pronounced social biases. Clarifying how reasoning gains interact with bias mitigation-and whether the two objectives inherently trade off-therefore remains an open and pressing research problem. Our study begins by benchmarking three bias-mitigation strategies-supervised fine-uning (SFT), knowledge distillation (KD), and rule-based reinforcement learning (RL)-under identical conditions, establishing their baseline strengths and weaknesses. Building on these results, we vary the proportion of debias-focused and reasoning-centric samples within each paradigm to chart the reasoning-versus-bias trade-off. Our sweeps reveal a consistent sweet spot: a roughly 1:4 mix trained with reinforcement learning cuts stereotype scores by 10% while retaining 88% of the model's original reasoning accuracy, offering concrete guidance for balancing fairness and capability in MLLMs.

Figures

Figures reproduced from arXiv: 2507.23067 by the authors.

Figure 1
Figure 1. Qwen2.5-vl-7B sweet spot between reasoning and bias [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model Performance Comparison Across Training Schemes.GRPO consistently improves bias mitigation, boosting Qwen3- 8B’s ambiguous score from 0.79 to 0.87 on BBQ, and raising Qwen2.5-VL and Intern3VL from 0.35/0.38 to 0.54 on VLBiasBench, outperforming SFT and Distillation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The Sweet Spot Between Reasoning and Bias.. Varying the ratio of debiasing data reveals a consistent trade-off curve. Across models, 10–20% debiasing yields the best balance—e.g., GRPO on Qwen3-8B reduces bias by 14.2% with minimal reasoning loss. mitigation capabilities and minor degeneration in reasoning. After evaluating our trained models with the same scal￾ing strategies, we normalize the test results and plot … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 21 canonical work pages

  1. [1]

    Clip the bias: How useful is balancing data in multimodal learning?,

    Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D’Amour, and Xiaohua Zhai. Clip the bias: How useful is balancing data in multimodal learning?,

    work page

  2. [2]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. 2

    2025

  3. [3]

    A prompt array keeps the bias away: Debiasing vision-language models with ad- versarial learning

    Hugo Berg, Siobhan Hall, Yash Bhalgat, Hannah Kirk, Alek- sandar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing vision-language models with ad- versarial learning. InProceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Compu- tational Linguistics and the 12th International Joint Con- ference on Nat...

    work page 2022

  4. [4]

    Bow- man, Julian Michael, Ethan Perez, and Miles Turpin

    James Chua, Edward Rees, Hunar Batra, Samuel R. Bow- man, Julian Michael, Ethan Perez, and Miles Turpin. Bias- augmented consistency training reduces biased reasoning in chain-of-thought, 2025. 2

    work page 2025

  5. [5]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025. 3

    work page 2025

  6. [6]

    Open r1: A fully open reproduction of deepseek-r1, 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, 2025. 2

    work page 2025

  7. [7]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025. 1, 2

    work page 2025

  8. [8]

    Sanchit Kabra, Akshita Jha, and Chandan K. Reddy. Rea- soning towards fairness: Mitigating bias in language models through reasoning-guided fine-tuning, 2025. 1, 2

    work page 2025

Show all 34 references
  1. [9]

    Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems.arXiv preprint arXiv:2404.09486,

    Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma. Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems.arXiv preprint arXiv:2404.09486,

    arXiv

  2. [10]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Ed- wards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. 3

    2023 arXiv

  3. [11]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning. InThe 59th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), 2021. 3

    2021

  4. [12]

    Metaspatial: Reinforcing 3d spa- tial reasoning in vlms for the metaverse.arXiv preprint arXiv:2503.18470, 2025

    Zhenyu Pan and Han Liu. Metaspatial: Reinforcing 3d spa- tial reasoning in vlms for the metaverse.arXiv preprint arXiv:2503.18470, 2025. 1

    2025

  5. [13]

    Codev- bench: How do llms understand developer-centric code com- pletion?arXiv preprint arXiv:2410.01353, 2024

    Zhenyu Pan, Rongyu Cao, Yongchang Cao, Yingwei Ma, Binhua Li, Fei Huang, Han Liu, and Yongbin Li. Codev- bench: How do llms understand developer-centric code com- pletion?arXiv preprint arXiv:2410.01353, 2024. 1

    2024 arXiv

  6. [14]

    Chain-of-action: Faithful and multimodal question an- swering through large language models.arXiv preprint arXiv:2403.17359, 2024

    Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. Chain-of-action: Faithful and multimodal question an- swering through large language models.arXiv preprint arXiv:2403.17359, 2024. 1

    2024 arXiv

  7. [15]

    Conv-coa: Improving open-domain question answering in large language models via conversational chain-of-action

    Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. Conv-coa: Improving open-domain question answering in large language models via conversational chain-of-action. arXiv preprint arXiv:2405.17822, 2024. 1

    2024 arXiv

  8. [16]

    Do code llms understand design patterns? In2025 IEEE/ACM Inter- national Workshop on Large Language Models for Code (LLM4Code), pages 209–212

    Zhenyu Pan, Xuefeng Song, Yunkun Wang, Rongyu Cao, Binhua Li, Yongbin Li, and Han Liu. Do code llms understand design patterns? In2025 IEEE/ACM Inter- national Workshop on Large Language Models for Code (LLM4Code), pages 209–212. IEEE, 2025. 1

    2025

  9. [17]

    BBQ: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Pad- makumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dub...

    2022

  10. [18]

    Strengthening multi- modal large language model with bootstrapped preference optimization, 2024

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Run- tao Liu, Rui Pan, and Tong Zhang. Strengthening multi- modal large language model with bootstrapped preference optimization, 2024. 2

    2024

  11. [19]

    Sciscigpt: Advancing human- ai collaboration in the science of science.arXiv preprint arXiv:2504.05559, 2025

    Erzhuo Shao, Yifang Wang, Yifan Qian, Zhenyu Pan, Han Liu, and Dashun Wang. Sciscigpt: Advancing human- ai collaboration in the science of science.arXiv preprint arXiv:2504.05559, 2025. 1

    2025

  12. [20]

    Math- llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math- llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024. 2

    2024

  13. [21]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. 2

    2025

  14. [22]

    Llamav-o1: Rethinking step-by- step visual reasoning in llms, 2025

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Llamav-o1: Rethinking step-by- step visual reas...

    2025

  15. [23]

    Jacob-Junqi Tian, Omkar Dige, D. B. Emerson, and Faiza Khan Khattak. On the role of reasoning in the iden- tification of subtle stereotypes in natural language, 2024. 2

    2024

  16. [24]

    Vlbiasbench: A com- prehensive benchmark for evaluating bias in large vision- language model.arXiv preprint arXiv:2406.14194, 2024

    Sibo Wang, Xiangkui Cao, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen, and Wen Gao. Vlbiasbench: A com- prehensive benchmark for evaluating bias in large vision- language model.arXiv preprint arXiv:2406.14194, 2024. 3

    2024 arXiv

  17. [25]

    Zecheng Wang, Xinye Li, Zhanyue Qin, Chunshan Li, Zhiy- ing Tu, Dianhui Chu, and Dianbo Sui. Can we debias mul- timodal large language models via model editing? InPro- ceedings of the 32nd ACM International Conference on Mul- timedia, page 3219–3228, New York, NY , USA, 2024. ...

    2024

  18. [26]

    Does reasoning introduce bias? a study of social bias evaluation and mitigation in llm reason- ing, 2025

    Xuyang Wu, Jinming Nian, Ting-Ruen Wei, Zhiqiang Tao, Hsin-Tai Wu, and Yi Fang. Does reasoning introduce bias? a study of social bias evaluation and mitigation in llm reason- ing, 2025. 1 5

    2025

  19. [27]

    Llava-cot: Let vision language models reason step- by-step, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step- by-step, 2024. 2

    2024

  20. [28]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  21. [29]

    Benchmarking trustworthiness of multi- modal large language models: A comprehensive study.arXiv preprint arXiv:2406.07057, 2024

    Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, et al. Benchmarking trustworthiness of multi- modal large language models: A comprehensive study.arXiv preprint arXiv:2406.07057, 2024. 2

    2024 arXiv

  22. [30]

    VLStereoSet: A study of stereotypical bias in pre-trained vision-language models

    Kankan Zhou, Eason Lai, and Jing Jiang. VLStereoSet: A study of stereotypical bias in pre-trained vision-language models. InProceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Confer...

    2022

  23. [31]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze L...

    2025

  24. [32]

    For SFT and model distillation, we utilize theLLaMA-Factoryframework with hyperparameter configurations listed in Table 1

    Training Hyperparameters In this section, we provide details of the framework and hy- perparameter settings used for training. For SFT and model distillation, we utilize theLLaMA-Factoryframework with hyperparameter configurations listed in Table 1. For GRPO, we utilize theEas...

  25. [33]

    Test Results We list the results for our test results in Table 3 and Table 4

  26. [34]

    1 Table 3

    Sampling Strategy For all the evaluations in our study, we use a random seed of 42, a maximum response token limit of 10,000, and gener- ate 5 responses per prompt, with instructions for the models to enclose their final answers in\boxed{}. 1 Table 3. Performance under Distill...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.