Pith. sign in

REVIEW 5 major objections 5 minor 116 references

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark scores AI creativity by counting how many tries it takes to match a human-level joke.

desk verdict LoTbench is a serious attempt at creative evaluation, but its score tracks convergence to a cued answer; the MMMU correlation is about reasoning, not creativity. read the letter →

arxiv 2501.15147 v2 pith:BJUHOX7H submitted 2025-01-25 cs.AI cs.HC

Zhongzhan Huang , Shanshan Zhong , Pan Zhou , Shanghua Gao , Marinka Zitnik , Liang Lin This is my paper · ORCID
classification cs.AIcs.HC
keywords creativityevaluationmultimodallargelanguagemodelsLeap-of-ThoughtOogirigamecausalinterventionhumorgenerationMMMUcorrelationbenchmarkdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Creativity is hard to benchmark because a good answer can be one of many, and standard multiple-choice tests can leak the creative leap instead of measuring it. This paper argues that creativity can instead be measured as the cost of getting there: how many rounds of questioning, clues, and rethinking a multimodal LLM needs before it produces a response that a causal evaluator judges to be a genuinely different but equally good version of a human-level creative answer. The platform is Oogiri, a Japanese humor game that asks players to answer images, text, or both with unexpected funny lines, and the benchmark, LoTbench, scores a model by the average number of rounds needed. The paper reports that most current models sit near the lower band of human creativity, that the best models approach typical human performance, and that LoTbench scores correlate strongly with a general multimodal cognition benchmark (MMMU), suggesting cognition underlies early-stage creativity.

What carries the argument

The load-bearing object is DAESO ('different approach but equally satisfactory outcome'), a criterion for judging that a model's fill-in-the-blank answer is as creative as the human reference even though its wording differs. E1, a text-based LLM, applies the criterion by reconstructing, in text space, the causal chain that connects the image caption to the HHCR, then performing a do-intervention that replaces the HHCR's key text with the model's answer and checking that the final functional node of the chain is unchanged. A second evaluator, E2, answers the model's spontaneous yes/no questions, and the round count feeds an exponential score. The masked-language-model task keeps responses comparable while still leaving room for creative leaps.

What would settle it

Take the 106 LoTbench samples, have human judges rate whether each model answer that E1 accepted is as creative as the human reference, and compare the agreement rate with the rate for answers E1 rejected; if the two rates are statistically indistinguishable, the causal evaluator is not tracking human creativity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a way to turn 'creativity' from a subjective quality into a measurable interaction cost. Given an image, its caption, and a carefully authored high-quality human-level creative response (HHCR) with its key word masked, the model under test must fill the mask; if its answer does not pass the DAESO test—different approach but equally satisfactory outcome—it gets to ask a yes/no question, receive a clue, and try again. The model's creativity score is an exponentially decaying function of the number of rounds it needs. Using this framework on 106 curated HHCRs, the paper finds that most multimodal LLMs cluster near the low human band, the best models approach the medium-high human band, and the ranking correlates strongly with MMMU but only weakly with standard selection and ranking creativity metrics. The paper further claims this interaction sequence visualizes the model's creative thought process rather than merely scoring its final answer.

Load-bearing premise

The benchmark's validity rests on the 106 human-level creative references being genuinely human-level creative and on the LLM evaluator reliably recognizing when a different answer is just as creative.

Editorial extensions

If this is right

  • Creativity of a multimodal LLM can be quantified without judging the quality of a single open-ended response, by measuring how quickly the model converges to a human-level creative answer through interaction.
  • The finding that LoTbench scores correlate strongly with MMMU while standard creativity metrics do not implies that general multimodal cognition—perception, knowledge, and reasoning—is a foundation for early-stage creative association.
  • Because chain-of-thought prompting does not reliably raise LoTbench scores, improving logical reasoning alone is not enough to make models more creative.
  • The close ranking of strong LLMs with human participants implies that current models are not far from average human creativity and may surpass it if their generation can be stimulated.
  • Information leakage is mitigated because the model writes its own responses, so the test data cannot be memorized in advance from the internet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the MMMU–LoTbench correlation reflects a real cognitive dependency, then training that strengthens perception, knowledge integration, and relational reasoning should improve creativity scores more than training that merely sharpens step-by-step logic.
  • The DAESO criterion is a candidate formalization of creative equivalence that could be calibrated further: a larger human study could measure how often humans agree with E1's accept or reject decisions, and disagreement rates could be used to retune the criterion.
  • The round-count metric is task-agnostic in principle; it could be ported to other open-ended creative domains, such as scientific hypothesis generation or product design, wherever human-level reference responses can be authored.
  • Because the benchmark depends on one LLM evaluator, replacing E1 with a panel or with human oversight would be the natural robustness test; if rankings shift, the metric is partly measuring the evaluator's tastes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a two-part evaluation pipeline for creativity in multimodal LLMs. First, it presents Oogiri-GO, a multilingual and multimodal Oogiri humor dataset with over 130,000 samples, and uses it for standard choice and ranking evaluations, reporting that existing MLLMs perform weakly and that the authors' CLoTv2 fine-tuning improves results. Second, it proposes LoTbench, an interactive benchmark in which a model fills a masked key text in a response and is judged by a causal evaluator E1 for whether the response is a 'different approach but equally satisfactory outcome' (DAESO) relative to a single human-level creative response (HHCR); an evaluator E2 answers the model's Yes/No questions, clues are inserted every five rounds, and the creativity score is the exponentially decaying function of the number of rounds needed to reach the HHCR. The paper reports that most LLMs show constrained creativity, that the human–LLM gap is not large, and that LoTbench scores correlate strongly with MMMU (Pearson 0.801) while standard creativity evaluations do not.

Significance. If the benchmark's validity holds, this would be a useful contribution: Oogiri-GO is a large, publicly released resource; LoTbench addresses real limitations of static choice/ranking evaluations (information leakage, interpretability); and the attempt to formalize DAESO through causal chains is an interesting idea. The paper also ships a project page, includes extensive model comparisons, and reports a user study, which are tangible assets. However, the central claims rest on several assumptions that are not independently verified: the HHCR references are partly generated by the authors' own model, the causal equations are never actually computed, the E1 evaluator is validated on only 43 samples, and the interactive task may measure convergence to a cued answer rather than open-ended creativity. These issues are load-bearing for the headline conclusions, so the paper needs substantive revision before the benchmark can be accepted as a valid creativity metric.

major comments (5)
  1. [§5.1, §5.3, §5.5.1] The central measure is the number of rounds needed to reach a single pre-specified HHCR R (Alg. 1; Eq. (8)), and the MLM task fixes all text except the annotated key text κ (Sec. 5.3). E1's DAESO criterion (1) requires Rt and R to share the same creative point, expressed in Eq. (6) as both mapping to the same node C(j), and Eq. (7) requires the final functional node to be preserved. Consequently, any response that is creative in a different, equally valid direction is scored as a miss, and the clues (every 5 rounds) and E2's Yes/No answers are both derived from R. The round count therefore measures how quickly the model converges to one intended answer under an oracle regime; this is a convergent recognition/reasoning task, not open-ended divergent creativity. This threat is not addressed in the limitation section (Sec. 7.5).
  2. [§5.5.2–§5.5.3, §7.2] The causal equations (3)–(7) are never computed. E1 is implemented by prompting a text LLM to describe and judge the causal chains in language (Sec. 5.5.3), and the do(κ(R) → κ(Rt)) intervention is approximated by a semantic replacement in the prompt; Eq. (7)'s probability change ΔP is not estimated. The only validation of this evaluator is a manual check on 43 DAESO samples (Sec. 7.2), with no inter-annotator agreement or error analysis beyond the accuracy bars in Fig. 15. Because every LoTbench score in Fig. 10 passes through E1, the benchmark's quantitative claims rest on this unvalidated proxy.
  3. [§5.2, §5.3] The HHCR references used as the LoTbench target are not independent human references: Sec. 5.3 states that the 106 samples were curated 'with Oogiri-GO and the help of CLoTv2,' and CLoTv2 is the authors' own model trained in Sec. 5.2 on Oogiri-GO. No independent human-quality validation of these HHCRs is reported. If these references are not broadly accepted as being at human creative level, the 'human-level creativity' target, and hence the score, is not grounded.
  4. [§6.2] The human comparison in Sec. 6.2 reports 21 subjects but says they were divided into three equal groups 'each group containing 9 individuals'; this is arithmetically inconsistent, and the human high/medium/low reference points in Fig. 10 are not accompanied by any uncertainty or statistical test. The claim that the human–LLM gap is 'not insurmountable' therefore lacks a quantitative basis.
  5. [§7.4, Fig. 16(b)] The strong correlation with MMMU (Pearson 0.801) is interpreted as evidence that LoTbench aligns with cognitive theories, but the task design already rewards general reasoning: the model receives E2 Yes/No answers and clue sets derived from R, so faster convergence can reflect deductive search ability under partial information rather than creativity. The correlation analysis does not control for this confounding factor, so the claim that LoTbench measures creativity better than standard evaluation is not established by Fig. 16.
minor comments (5)
  1. [§7.1] The section title 'Ohter Types of Evaluation' contains a typo; it should read 'Other Types of Evaluation.'
  2. [§5.7, Eq. (8)] The hyperparameters αc and βc are set to 0.2 and 1.0, respectively, but no sensitivity analysis is reported; βc is a multiplicative constant that does not affect model rankings, and the exponential form of the score is not justified.
  3. [§4] The notation 'mTn' is used to denote choice questions, but the relationship between the 'm' and 'n' in the notation and the task names (2T1, 3T1, 4T1, 5T2) is not defined explicitly in the main text.
  4. [§5.5.3] The text says explicitly constructing the causal chain is 'highly challenging' and then delegates the construction to a text LLM; this is a reasonable engineering choice, but it should be stated more clearly that the causal formalism in Sec. 5.5.2 is a motivating illustration rather than an implemented procedure.
  5. [§7.2, Fig. 15] The left panel of Fig. 15 reports 'success rates' for causal construction and intervention, but the definition of a successful construction or intervention is not given; a precise scoring rubric is needed for reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

LoTbench's score reduces by construction to convergence to a single pre-annotated HHCR; the HHCR ground truth is partly synthesized by the authors' own CLoTv2.

  1. self definitional [Sec. 5.5.1–5.5.2 and Sec. 5.7 (Eq. 8)]
    "According to criterion (1), if Rt and R are DAESO, for Eq. (3), there also exists an i′ ≤ last(Rt) such that f (R(i′)t ) → C(j). (6) ... The fewer rounds required to reach HHCRs, the more creative the LLM is deemed to be, and vice versa."

    The creativity score Sc is defined by Eq. (8) as an exponential function of the round at which E1 first returns True in Algorithm 1. But E1's DAESO criterion is Eq. (6): Rt must map to the same caption-chain node C(j) as the HHCR R through the same creative interpretation f. Thus 'reaching the HHCR level' is implemented as 'sharing the pre-annotated creative point of the single reference R'. A response that is creative in a different direction is scored as a miss by construction. The measured variable is therefore convergence time to a fixed target reference, not an independent measurement of open-ended divergent creativity.

  2. other [Sec. 5.3 (Data Construction in LoTbench)]
    "we carefully and manually curated 106 HHCR samples suited for LoTbench, with Oogiri-GO and the help of CLoTv2 trained in Section 5.2 to generate brand-new HHCRs that meet MLM requirements."

    LoTbench defines creativity as the cost to reach a high-quality human-level creative response R, and R is the target against which every model is scored. A portion of these target HHCRs are generated by CLoTv2, a model fine-tuned by the same authors on the Oogiri-GO training data. The reference anchor is therefore not fully independent: models whose outputs resemble the authors' own tuned model's generation style will, by construction, be scored as closer to 'human-level creativity'. The MMMU correlation provides some external anchoring, but it does not remove the self-referential component in the ground-truth targets themselves.

full rationale

The central LoTbench claim reduces, by the paper's own definitions, to a convergence-to-reference score. Algorithm 1 stops at the first round where E1(Rt, R) is True, and Eq. (8) maps that round count to the creativity score. E1's DAESO criterion, Eq. (6), requires Rt to map to the same caption-chain node C(j) as the fixed HHCR R through the same creative explanation f. So a model is scored as creative exactly when it converges to the pre-specified creative point of one reference response, and any equally valid but different creative direction is definitionally a miss. This is a self-definitional reduction: the output variable is the time to match an input reference, not an independent estimate of divergent creativity. The target HHCRs are also partly synthesized by the authors' CLoTv2 model, which further weakens the independence of the ground truth. The human-subject comparison and the MMMU correlation in Sec. 7.4 provide some external validation and prevent the score from being higher, but they do not change the fact that the primary metric is equivalent to convergence to a fixed R by construction. The 43-sample E1 validation and the use of GPT-4o mini as the evaluator are real validity risks, but they are correctness concerns rather than additional circularity. The standard Oogiri-GO evaluation in Sec. 6 is a separate, largely non-circular contribution.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The LoTbench score rests on several choices the paper does not justify: the hand-set decay parameters in the score formula, the assumption that Oogiri likes measure creativity, the ad hoc causal-chain representation of creativity, the use of CLoTv2-generated HHCRs as human-level references, and the reliance on a proprietary LLM to implement causal intervention. The MMMU correlation is the main external anchor.

free parameters (7)
  • αc (decay constant in creativity score) = 0.2
    Eq. (8) uses exp(-αc*t); the paper sets 0.2 without sensitivity analysis. This choice directly determines the score range and relative gaps between models.
  • βc (scale in creativity score) = 1.0
    Eq. (8) scale factor; chosen by hand, harmless to ranking but part of the formula.
  • α, β (weak-association sampling thresholds) = 25, 70
    Sec 5.2.2 removes top 25% and bottom 70% of similarity-ranked conditions to define weakly-associated conditions; these affect which HHCRs are generated and thus the LoTbench ground truth.
  • n (candidate conditions per sample) = 100
    Sec 5.2.2 samples 100 candidate conditions; a computational budget choice that shapes condition diversity.
  • ρc (empty-condition probability) = not specified
    Sec 5.2.1 sets condition empty with probability ρc, but the value is not reported; affects training distribution.
  • clue interval = 5 rounds
    Sec 5.3: substantive clue added every 5 rounds in LoTbench; affects how quickly models approach HHCR and thus Sc.
  • m (repeated evaluations) and N (max rounds) = m=3, N=15
    Sec 5.7 sets m=3; Sec 6.2 implies N=15; both bound the score and interact with αc.
assumptions (5)
  • domain assumption Oogiri response popularity (likes) is a valid proxy for creativity
    Oogiri-GO uses likes as human preference annotations (Sec 3.1), and standard evaluation/ranking treats them as ground truth.
  • ad hoc to paper Creativity of a response can be represented as a causal chain with a single mapping f from a response node to a caption node and matching final functions
    Eqs. (3)-(4) in Sec 5.5.2 define this representation without evidence that creative humor decomposes this way.
  • ad hoc to paper A text LLM's description of causal chains faithfully implements the causal intervention do(κ(R)->κ(Rt))
    Sec 5.5.3 replaces the formal equations with an LLM prompt; validity rests on the LLM, not a causal estimator.
  • domain assumption HHCRs synthesized by CLoTv2 and manually curated are human-level creative responses
    Sec 5.3 constructs LoTbench ground truth using CLoTv2; no independent human rating of these 106 HHCRs is reported.
  • domain assumption MMMU scores measure human-like cognition and therefore the LoTbench-MMMU correlation indicates alignment with cognitive theories
    Sec 7.4 uses a single correlation (r=0.801) to claim cognitive alignment; MMMU's construct validity is assumed.
invented entities (1)
  • DAESO (different approach but equally satisfactory outcome)
    purpose: Defines when a model's response has creativity equivalent to the reference HHCR, operationalized by the E1 LLM judge.
    Introduced in Sec 5.5.1; no external falsifiable handle, only an LLM judgment validated on 43 self-collected examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/BJUHOX7H

@misc{pith2026250115147,
  author       = {Pith},
  title        = {Pith review of: A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BJUHOX7H}},
  note         = {Machine review of arXiv:2501.15147}
}
read the original abstract

Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game, a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. https://lotbench.github.io

Figures

Figures reproduced from arXiv: 2501.15147 by the authors.

Figure 1
Figure 1. Leap-of-Thought (LoT) for creativity. (a) Comparison of CoT and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The motivation of different paradigms to measure creativity. (Left) Standard Evaluation: Assess LLMs by designing selection and ranking [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview of proposed interactive creativity evaluation LoT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The main task in LoTbench is masked language modeling (MLM). [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The details of LoT-oriented instructions templates. We take “Image to Text” as an example, see the Appendix of the conference version [1] [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The LoT-oriented instruction templates. evaluator E1. Intuitively, fewer rounds imply statistically higher creativity for the LLM. Throughout this process, the LLM can continuously ask questions about R in each round, and the system E2 will respond with Yes/No. This re…
Figure 7
Figure 7. Figure 7: The overview of CLoTv2 to tune LLM for data synthesis. Left: CLoTv2 relies on two LoT-boosting stages, including associable instruction [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The overview of DAESO. (a) The difference between semantic similarity and functional similarity. (b) The mathematical modeling for DAESO [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The accuracy (%) of choice questions and the NDCG (%) of ranking questions on our CLoT and various reasoning frameworks. The baseline is Qwen-VL on multilingual I2T task. For mTn choice questions, one needs to select n correct answers from m options. non-multilingual m…
Figure 11
Figure 11. Figure 11: Specific creative responses. We visualize the outputs of the [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Example of visualization for creative thinking in LoTbench. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 10
Figure 10. Figure 10: The ranking results of LLM’s creativity by LoTbench. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 14
Figure 14. Figure 14: The comparison between human preference and LLM-as-a [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: Analysis of the discriminative abilities of [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: The correlation analysis between LLM cognition benchmark MMMU and the proposed creatvity benchmark. (a) The correlation between [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

116 extracted references · 32 canonical work pages

  1. [1]

    Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation,

    S. Zhong, Z. Huang, S. Gao, W. Wen, L. Lin, M. Zitnik, and P . Zhou, “Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 246–13 257

    2024

  2. [2]

    Qwen technical report,

    J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P . Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  4. [4]

    Finetuned language models are zero- shot learners,

    J. Wei, M. Bosma, V . Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero- shot learners,” arXiv preprint arXiv:2109.01652, 2021

    arXiv 2021

  5. [5]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought,

    A. Saparov and H. He, “Language models are greedy reasoners: A systematic formal analysis of chain-of-thought,” arXiv preprint arXiv:2210.01240, 2022

    arXiv 2022

  6. [6]

    Socratic models: Composing zero-shot multimodal reasoning with language,

    A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani et al., “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598, 2022

    arXiv 2022

  7. [7]

    Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models,

    S. Zhong, Z. Huang, W. Wen, J. Qin, and L. Lin, “Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 567–578

    2023

  8. [8]

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks,

    T. Guo, B. Nan, Z. Liang, Z. Guo, N. Chawla, O. Wiest, X. Zhang et al. , “What can large language models do in chemistry? a comprehensive benchmark on eight tasks,” Advances in Neural Information Processing Systems, vol. 36, pp. 59 662–59 688, 2023

    2023

Show all 116 references
  1. [9]

    Scemqa: A scientific college entrance level multimodal question answering benchmark,

    Z. Liang, K. Guo, G. Liu, T. Guo, Y. Zhou, T. Yang, J. Jiao, R. Pi, J. Zhang, and X. Zhang, “Scemqa: A scientific college entrance level multimodal question answering benchmark,” arXiv preprint arXiv:2402.05138, 2024

    2024 arXiv

  2. [10]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P . Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P . Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” Advances in Neural Information Processing Systems, vol. 35, pp. 2507–2521, 2022

    2022

  3. [11]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun et al. , “Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  4. [12]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  5. [13]

    A survey on benchmarks of multimodal large language models,

    J. Li and W. Lu, “A survey on benchmarks of multimodal large language models,” arXiv preprint arXiv:2408.08632, 2024. 15

    2024 arXiv

  6. [14]

    A survey on evaluation of large language models,

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024

    2024

  7. [15]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  8. [16]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  9. [17]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  10. [18]

    Tree of thoughts: Deliberate problem solv- ing with large language models, may 2023,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solv- ing with large language models, may 2023,” arXiv preprint arXiv:2305.10601, 2023

    2023 arXiv

  11. [19]

    Large language model guided tree-of-thought,

    J. Long, “Large language model guided tree-of-thought,” arXiv preprint arXiv:2305.08291, 2023

    2023 arXiv

  12. [20]

    Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge,

    A. Talmor, O. Tafjord, P . Clark, Y. Goldberg, and J. Berant, “Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge,” Advances in Neural Information Processing Systems, vol. 33, pp. 20 227–20 237, 2020

    2020

  13. [21]

    Cognitive science: Leap of thought,

    E. Callaway, “Cognitive science: Leap of thought,” 2013

    2013

  14. [22]

    Mental leaps: analogy in creative thought,

    K. J. Holyoak, P . Thagard, and S. Sutherland, “Mental leaps: analogy in creative thought,” Nature, vol. 373, no. 6515, pp. 572– 572, 1995

    1995

  15. [23]

    The leap of thinking: A comparison of heidegger and the zen master dogen,

    C. Olson, “The leap of thinking: A comparison of heidegger and the zen master dogen,” Philosophy Today, vol. 25, no. 1, p. 55, 1981

    1981

  16. [24]

    A review of mental leaps: analogy in creative thought,

    D. Hofstadter, “A review of mental leaps: analogy in creative thought,” AI Magazine, vol. 16, no. 3, pp. 75–75, 1995

    1995

  17. [25]

    K. J. Holyoak and P . Thagard, Mental leaps: Analogy in creative thought. MIT press, 1996

    1996

  18. [26]

    Measuring creative thinking: An activity-based approach,

    J. Kitto, D. Lok, and E. Rudowicz, “Measuring creative thinking: An activity-based approach,” Creativity Research Journal , vol. 7, no. 1, pp. 59–69, 1994

    1994

  19. [27]

    Eeg complexity and performance measures of creative thinking,

    M. M ¨olle, L. Marshall, B. Wolf, H. L. Fehm, and J. Born, “Eeg complexity and performance measures of creative thinking,” Psychophysiology, vol. 36, no. 1, pp. 95–104, 1999

    1999

  20. [28]

    Development and validation of team creativity measures: A complex systems perspective,

    H. Jiang and Q.-p. Zhang, “Development and validation of team creativity measures: A complex systems perspective,” Creativity and Innovation Management, vol. 23, no. 3, pp. 264–275, 2014

    2014

  21. [29]

    Kahneman, Thinking, fast and slow

    D. Kahneman, Thinking, fast and slow. macmillan, 2011

    2011

  22. [30]

    Glossary of owarai terms,

    Wikimedia, “Glossary of owarai terms,” https://en.wikiped ia.org/wiki/Glossary of owarai terms, 2023. [Online]. Available: https://en.wikipedia.org/wiki/Glossary of owarai terms

    2023

  23. [31]

    Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge,

    B. Y. Lin, Z. Wu, Y. Yang, D.-H. Lee, and X. Ren, “Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge,” arXiv preprint arXiv:2101.00376 , 2021

    2021 arXiv

  24. [32]

    Brainteaser: Lateral thinking puz- zles for large language model,

    Y. Jiang, F. Ilievski, and K. Ma, “Brainteaser: Lateral thinking puz- zles for large language model,” arXiv preprint arXiv:2310.05057 , 2023

    2023 arXiv

  25. [33]

    Birdqa: A bilingual dataset for question answering on tricky riddles,

    Y. Zhang and X. Wan, “Birdqa: A bilingual dataset for question answering on tricky riddles,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 10, 2022, pp. 11 748–11 756

    2022

  26. [34]

    Cogvlm: Visual expert for pretrained language models,

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, J. Xu, B. Xu, J. Li, Y. Dong, M. Ding, and J. Tang, “Cogvlm: Visual expert for pretrained language models,” arXiv preprint arXiv: 2311.03079, 2023

    2023 arXiv

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P . Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P . Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  28. [36]

    Weak-eval- strong: Evaluating and eliciting lateral thinking of llms with situation puzzles,

    Q. Chen, B. Zhang, G. Wang, and Q. Wu, “Weak-eval- strong: Evaluating and eliciting lateral thinking of llms with situation puzzles,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.06733

    2024 arXiv

  29. [37]

    Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles,

    S. Huang, S. Ma, Y. Li, M. Huang, W. Zou, W. Zhang, and H.-T. Zheng, “Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles,” arXiv preprint arXiv:2308.10855, 2023

    2023 arXiv

  30. [38]

    The effect of individual differences in cognitive style and motives in solving insight problems,

    O. Martinsen, “The effect of individual differences in cognitive style and motives in solving insight problems,” Scandinavian Journal of Educational Research, vol. 38, no. 2, pp. 83–96, 1994

    1994

  31. [39]

    Insight problems revisited: The influence of cognitive styles and experience on creative problem solving,

    ——, “Insight problems revisited: The influence of cognitive styles and experience on creative problem solving,” Creativity Research Journal, vol. 6, no. 4, pp. 435–447, 1993

    1993

  32. [40]

    The explorer and the assimilator: A cognitive style distinction and its potential implications for innovative problem solving,

    G. Kaufmann, “The explorer and the assimilator: A cognitive style distinction and its potential implications for innovative problem solving,” Scandinavian Journal of Educational Research , vol. 23, no. 3, pp. 101–108, 1979

    1979

  33. [41]

    Cognition and creativity,

    M. A. Runco and I. Chand, “Cognition and creativity,” Educational psychology review, vol. 7, pp. 243–267, 1995

    1995

  34. [42]

    The associative basis of the creative process

    S. Mednick, “The associative basis of the creative process.” Psy- chological review, vol. 69, no. 3, p. 220, 1962

    1962

  35. [43]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  36. [44]

    Minigpt-v2: large language model as a unified interface for vision-language multi- task learning,

    J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P . Zhang, R. Krishnamoor- thi, V . Chandra, Y. Xiong, and M. Elhoseiny, “Minigpt-v2: large language model as a unified interface for vision-language multi- task learning,” arXiv preprint arXiv:2310.09478, 2023

    arXiv

  37. [45]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P . Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  38. [46]

    Probing the creativity of large language models: Can models produce divergent semantic association?

    H. Chen and N. Ding, “Probing the creativity of large language models: Can models produce divergent semantic association?” arXiv preprint arXiv:2310.11158, 2023

    2023 arXiv

  39. [47]

    Unleashing the creative mind: Language model as hierarchical policy for improved exploration on challenging problem solving,

    Z. Ling, Y. Fang, X. Li, T. Mu, M. Lee, R. Pourreza, R. Memisevic, and H. Su, “Unleashing the creative mind: Language model as hierarchical policy for improved exploration on challenging problem solving,” arXiv preprint arXiv:2311.00694, 2023

    2023 arXiv

  40. [48]

    Brainstorm, then select: a generative language model improves its creativity score,

    D. Summers-Stay, C. R. Voss, and S. M. Lukin, “Brainstorm, then select: a generative language model improves its creativity score,” in The AAAI-23 Workshop on Creative AI Across Modalities , 2023

    2023

  41. [49]

    Papers and patents are becoming less disruptive over time,

    M. Park, E. Leahey, and R. J. Funk, “Papers and patents are becoming less disruptive over time,” Nature, vol. 613, no. 7942, pp. 138–144, 2023

    2023

  42. [50]

    Stiffness-aware neural net- work for learning hamiltonian systems,

    S. Liang, Z. Huang, and H. Zhang, “Stiffness-aware neural net- work for learning hamiltonian systems,” in International Confer- ence on Learning Representations, 2021

    2021

  43. [51]

    On fast simulation of dynamical system with neural vector enhanced numerical solver,

    Z. Huang, S. Liang, H. Zhang, H. Yang, and L. Lin, “On fast simulation of dynamical system with neural vector enhanced numerical solver,” Scientific Reports, vol. 13, no. 1, p. 15254, 2023

    2023

  44. [52]

    Story centaur: Large language model few shot learning as a creative writing tool,

    B. Swanson, K. Mathewson, B. Pietrzak, S. Chen, and M. Di- nalescu, “Story centaur: Large language model few shot learning as a creative writing tool,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demons...

    2021

  45. [53]

    Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing,

    T. Chakrabarty, V . Padmakumar, and H. He, “Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing,” arXiv preprint arXiv:2210.13669, 2022

    2022 arXiv

  46. [54]

    Promptchainer: Chaining large language model prompts through visual programming,

    T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai, “Promptchainer: Chaining large language model prompts through visual programming,” in CHI Conference on Human Factors in Computing Systems Extended Abstracts , 2022, pp. 1–10

    2022

  47. [55]

    Computational humor,

    K. Binsted, A. Nijholt, O. Stock, C. Strapparava, G. Ritchie, R. Ma- nurung, H. Pain, A. Waller, and D. O’Mara, “Computational humor,” IEEE intelligent systems, vol. 21, no. 2, pp. 59–69, 2006

    2006

  48. [56]

    ” a good pun is its own reword

    Z. Xu, S. Yuan, L. Chen, and D. Yang, “” a good pun is its own reword”: Can large language models understand puns?” arXiv preprint arXiv:2404.13599, 2024

    2024 arXiv

  49. [57]

    Inside jokes: Identifying humorous cartoon captions,

    D. Shahaf, E. Horvitz, and R. Mankoff, “Inside jokes: Identifying humorous cartoon captions,” in Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, 2015, pp. 1065–1074

    2015

  50. [58]

    Learning to evaluate humor in memes based on the incongruity theory,

    K. Tanaka, H. Yamane, Y. Mori, Y. Mukuta, and T. Harada, “Learning to evaluate humor in memes based on the incongruity theory,” in Proceedings of the Second Workshop on When Creative AI Meets Conversational AI, 2022, pp. 81–93

    2022

  51. [59]

    Hybrid multimodal fusion for humor detection,

    H. Xu, W. Liu, J. Liu, M. Li, Y. Feng, Y. Peng, Y. Shi, X. Sun, and M. Wang, “Hybrid multimodal fusion for humor detection,” in Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, 2022, pp. 15–21

    2022

  52. [60]

    A survey on approaches to com- putational humor generation,

    M. Amin and M. Burghardt, “A survey on approaches to com- putational humor generation,” in Proceedings of the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Her- itage, Social Sciences, Humanities and Literature , 2020, pp. 29–41. 16

    2020

  53. [61]

    Let’s be humor- ous: Knowledge enhanced humor generation,

    H. Zhang, D. Liu, J. Lv, and C. Luo, “Let’s be humor- ous: Knowledge enhanced humor generation,” arXiv preprint arXiv:2004.13317, 2020

    2004 arXiv

  54. [62]

    Stimulating creativity with funlines: A case study of humor generation in headlines,

    N. Hossain, J. Krumm, T. Sajed, and H. Kautz, “Stimulating creativity with funlines: A case study of humor generation in headlines,” arXiv preprint arXiv:2002.02031, 2020

    2002 arXiv

  55. [63]

    Does ai have a sense of humor? clef 2023 joker tasks 1, 2 and 3: using bloom, gpt, simplet5, and more for pun detection, location, interpretation and translation,

    O. Popova and P . Dadi ´c, “Does ai have a sense of humor? clef 2023 joker tasks 1, 2 and 3: using bloom, gpt, simplet5, and more for pun detection, location, interpretation and translation,” Proceedings of the Working Notes of CLEF , 2023

    2023

  56. [64]

    Mhadig: A multilingual humor-aided multiparty dialogue gen- eration in multimodal conversational setting,

    D. S. Chauhan, G. V . Singh, A. Ekbal, and P . Bhattacharyya, “Mhadig: A multilingual humor-aided multiparty dialogue gen- eration in multimodal conversational setting,” Knowledge-Based Systems, vol. 278, p. 110840, 2023

    2023

  57. [65]

    Mirror gradient: Towards robust multimodal recommender systems via exploring flat local minima,

    S. Zhong, Z. Huang, D. Li, W. Wen, J. Qin, and L. Lin, “Mirror gradient: Towards robust multimodal recommender systems via exploring flat local minima,” arXiv preprint arXiv:2402.11262 , 2024

    2024 arXiv

  58. [66]

    Visual programming for zero-shot open-vocabulary 3d visual ground- ing,

    Z. Yuan, J. Ren, C.-M. Feng, H. Zhao, S. Cui, and Z. Li, “Visual programming for zero-shot open-vocabulary 3d visual ground- ing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20 623–20 633

    2024

  59. [67]

    Towards revealing the mystery behind chain of thought: a theoretical perspective,

    G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang, “Towards revealing the mystery behind chain of thought: a theoretical perspective,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  60. [68]

    Faithful chain-of-thought reasoning,

    Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. Callison-Burch, “Faithful chain-of-thought reasoning,” arXiv preprint arXiv:2301.13379, 2023

    2023 arXiv

  61. [69]

    Dd- cot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models,

    G. Zheng, B. Yang, J. Tang, H.-Y. Zhou, and S. Yang, “Dd- cot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 5168–5191, 2023

    2023

  62. [70]

    Towards reasoning in large lan- guage models: A survey,

    J. Huang and K. C.-C. Chang, “Towards reasoning in large lan- guage models: A survey,” arXiv preprint arXiv:2212.10403, 2022

    2022 arXiv

  63. [71]

    Do androids laugh at electric sheep? Humor “understanding

    J. Hessel, A. Marasovi ´c, J. D. Hwang, L. Lee, J. Da, R. Zellers, R. Mankoff, and Y. Choi, “Do androids laugh at electric sheep? Humor “understanding” benchmarks from The New Yorker Cap- tion Contest,” in Proceedings of the ACL, 2023

    2023

  64. [72]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  65. [73]

    Cumulated gain-based evaluation of ir techniques,

    K. J ¨arvelin and J. Kek¨al¨ainen, “Cumulated gain-based evaluation of ir techniques,” ACM Transactions on Information Systems (TOIS), vol. 20, no. 4, pp. 422–446, 2002

    2002

  66. [74]

    Comparing the sensitivity of in- formation retrieval metrics,

    F. Radlinski and N. Craswell, “Comparing the sensitivity of in- formation retrieval metrics,” in Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, 2010, pp. 667–674

    2010

  67. [75]

    Sloane, The leader’s guide to lateral thinking skills: Powerful problem-solving techniques to ignite your team’s potential

    P . Sloane, The leader’s guide to lateral thinking skills: Powerful problem-solving techniques to ignite your team’s potential . Kogan Page Publishers, 2003

    2003

  68. [76]

    Creativity step by step,

    L. Thinking, “Creativity step by step,” By Edward de Bono, 1970

    1970

  69. [77]

    Mental leap,

    J. Lee, “Mental leap,” in Encyclopedia of the Sciences of Learning , N. M. Seel, Ed. Boston, MA: Springer US, 2012, pp. 2194–2194. [Online]. Available: https://doi.org/10.1007/978-1-4419-1428-6 1557

    2012 doi

  70. [78]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  71. [79]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  72. [80]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P . Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  73. [81]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821, 2021

    2021 arXiv

  74. [82]

    Will large-scale generative models corrupt future datasets?

    R. Hataya, H. Bao, and H. Arai, “Will large-scale generative models corrupt future datasets?” in ICCV, 2023

    2023

  75. [83]

    Model dementia: Generated data makes models forget,

    I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson, “Model dementia: Generated data makes models forget,” arXiv preprint arXiv:2305.17493, 2023

    2023 arXiv

  76. [84]

    Exploration of masked and causal language modelling for text generation,

    N. Micheletti, S. Belkadi, L. Han, and G. Nenadic, “Exploration of masked and causal language modelling for text generation,” arXiv preprint arXiv:2405.12630, 2024

    2024 arXiv

  77. [85]

    tinybenchmarks: evaluating llms with fewer examples,

    F. M. Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin, “tinybenchmarks: evaluating llms with fewer examples,” arXiv preprint arXiv:2402.14992, 2024

    2024 arXiv

  78. [86]

    metabench–a sparse benchmark to measure general ability in large language models,

    A. Kipnis, K. Voudouris, L. M. S. Buschoff, and E. Schulz, “metabench–a sparse benchmark to measure general ability in large language models,” arXiv preprint arXiv:2407.12844, 2024

    2024 arXiv

  79. [87]

    mplug-owl: Modularization empow- ers large language models with multimodality,

    Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P . Shi, Y. Shi et al., “mplug-owl: Modularization empow- ers large language models with multimodality,” arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  80. [88]

    Glm: General language model pretraining with autoregressive blank infilling,

    Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 320–335

    2022

  81. [89]

    Gemini: a family of highly capable multimodal mod- els,

    G. Team, “Gemini: a family of highly capable multimodal mod- els,” arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  82. [90]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. e. a. Chen, “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24 185–24 198

    2024

  83. [91]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  84. [92]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang et al., “Yi: Open foundation models by

  85. [93]

    ai,” arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  86. [94]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P . Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al. , “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  87. [95]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, and et al., “Instructblip: Towards general-purpose vision-language models with instruction tuning,” arXiv preprint arXiv:2305.06500, 2023

    2023 arXiv

  88. [96]

    Otter: A multi-modal model with in-context instruction tuning,

    B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,” arXiv preprint arXiv:2305.03726, 2023

    2023 arXiv

  89. [97]

    Baichuan 2: Open large-scale language models,

    Baichuan, “Baichuan 2: Open large-scale language models,” arXiv preprint arXiv:2309.10305 , 2023. [Online]. Available: https://arxiv.org/abs/2309.10305

    2023 arXiv

  90. [98]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  91. [99]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,” arXiv preprint arXiv:2303.16634, 2023

    2023 arXiv

  92. [100]

    Gpteval: A survey on assessments of chatgpt and gpt-4,

    R. Mao, G. Chen, X. Zhang, F. Guerin, and E. Cambria, “Gpteval: A survey on assessments of chatgpt and gpt-4,” arXiv preprint arXiv:2308.12488, 2023

    2023 arXiv

  93. [101]

    Mllm-bench, evaluating multi-modal llms using gpt-4v,

    W. Ge, S. Chen, G. Chen, J. Chen, Z. Chen, S. Yan, C. Zhu, Z. Lin, W. Xie, X. Wanget al., “Mllm-bench, evaluating multi-modal llms using gpt-4v,” arXiv preprint arXiv:2311.13951, 2023

    2023 arXiv

  94. [102]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  95. [103]

    A measure of similarity in textual data using spearman’s rank correlation coefficient,

    N. Arsov, M. Dukovski, B. Evkoski, and S. Cvetkovski, “A measure of similarity in textual data using spearman’s rank correlation coefficient,” arXiv preprint arXiv:1911.11750, 2019

    1911 arXiv

  96. [104]

    Investigating the catastrophic forgetting in multimodal large language models,

    Y. Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y. J. Lee, and Y. Ma, “Investigating the catastrophic forgetting in multimodal large language models,” arXiv preprint arXiv:2309.10313, 2023

    2023 arXiv

  97. [105]

    Quantifying creativity in art net- works,

    A. Elgammal and B. Saleh, “Quantifying creativity in art net- works,” arXiv preprint arXiv:1506.00711, 2015

    2015 arXiv

  98. [106]

    Devel- oping creativity in higher education for 21st century learners: A protocol for a scoping review,

    A. Egan, R. Maguire, L. Christophers, and B. Rooney, “Devel- oping creativity in higher education for 21st century learners: A protocol for a scoping review,” International Journal of educational research, vol. 82, pp. 21–27, 2017. 17 APPENDIX A: T HE DETAILS OF It AND GENERAT...

    2017

  99. [107]

    IMAGE: A given image

  100. [108]

    IMAGE CAPTION: A detailed description of the given image

  101. [109]

    You need to complete the part based on all the INPUT information

    RESPONSE: A sentence with a blank to be filled. You need to complete the part based on all the INPUT information

  102. [110]

    Carefully analyze and follow these hints to generate a creative and humorous

    TIPS: Some guidelines for the task, which include: • Q&A: Questions and answers related to . Carefully analyze and follow these hints to generate a creative and humorous . • CLUE: Descriptive hints about Understand and adhere to these clues to generate a creative and humorous ...

  103. [111]

    IMAGE: a given image

  104. [112]

    IMAGE CAPTION: a detailed description of the given image IMAGE

  105. [113]

    RESPONSE: a text of an IMAGE with the content ⟨WORD⟩ to be completed, you need to complete the ⟨WORD⟩ part according to IMAGE and IMAGE CAPTION

  106. [114]

    Q&A: some known queries and corresponding answers about ⟨WORD⟩

  107. [115]

    CLUE: some descriptive hints related to ⟨WORD⟩

  108. [116]

    WRONG-ANS: some innovative and humorous ⟨WORD⟩, you should not complete similar content In order to better complete the ⟨WORD⟩ in RESPONSE, so that the combination of IMAGE and RESPONSE is very humorous and creative, you can first use divergent thinking to ask a general questi...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.