Pith. sign in

REVIEW 4 major objections 5 minor 55 references

FLIP Reasoning Challenge

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that FLIP, a four-image story-ordering benchmark built from human consensus, exposes a large gap: best models reach 77.9% while people reach 95.3%.

desk verdict The FLIP dataset is a real addition to multimodal reasoning benchmarks, but the headline accuracies are measured on validation rather than held-out test, so the human-AI gap is not yet established. read the letter →

arxiv 2504.12256 v1 pith:IALPBXRR submitted 2025-04-16 cs.CV cs.AIcs.LG

Andreas Plesner , Turlan Kuzhagaliyev , Roger Wattenhofer This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords FLIPbenchmarkvisualreasoningsequentialstorytellingmultimodalAIimagecaptioningzero-shotevaluationcommon-sense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FLIP, a benchmark made from thousands of human-verified visual tasks on a blockchain identity platform, where a solver sees the same four images in two different orders and must pick the order that tells a coherent story. Its central claim is that this seemingly easy task exposes a large gap in current AI: the best open-sourced model answers correctly 75.5% of the time and the best closed-sourced model 77.9%, while the people who generated the tasks reach 95.3%. The paper also reports that feeding models written captions of the images works better than showing them the images directly, and that combining 15 models raises accuracy to 85.2%. If the benchmark measures what it claims, then no current system reasons about event sequences at a human level, and FLIP provides a concrete instrument for tracking progress toward that ability.

What carries the argument

The central object is the FLIP challenge: a pair of orderings, called stacks, of the same four images, where exactly one stack tells a meaningful story and the other does not; the correct answer comes from human consensus voting on the blockchain platform. The load-bearing mechanism in the evaluation is the caption pipeline: an image-captioning model (mainly the BLIP-2 family) turns each image into a text description, and a text-only reasoner then compares the two four-caption sequences and chooses the more coherent story. This separation lets the paper argue that contamination is unlikely, because captions restate each challenge in fresh text and the prompts never mention the source of the tasks.

What would settle it

Take a set of original FLIP challenges and an equal set of newly written flips that are structurally identical but have never been posted online; if models score much higher on the original set than on the new set, the reported gap is partly memorization rather than reasoning, while matching scores would support the paper's claim that FLIP measures genuine multimodal sequential reasoning.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that FLIP challenges—four images presented in two alternative orderings, with ground truth set by human consensus voting—are easy for people but hard for every tested model in zero-shot settings. With no training examples, the strongest open-sourced reasoning model reaches 75.5% and the strongest closed-sourced model 77.9%, compared with a 95.3% human accuracy; a logistic-regression ensemble of 15 models reaches 85.2%. The paper also finds that converting images to text captions helps: one top model improves from 69.6% when reading four raw input images to 75.2% when reasoning over captions, and several captioning-plus-reasoning combinations cross the 71% threshold the paper defines for a successful open-sourced solver. Task reframing gives small average gains, while providing models with historical examples of their own past answers lowers accuracy, and an error analysis shows the best models still fail on many strong-consensus flips.

Load-bearing premise

The benchmark's validity rests on the assumption that the publicly available FLIP challenges and the caption patterns have not been memorized by the tested models, so that solving them requires fresh reasoning rather than recall; the paper argues this is unlikely but does not directly test it against newly created flips.

Editorial extensions

If this is right

  • No tested model reaches the platform's human threshold of 92% accuracy, so FLIP currently separates machines from people on a task built by ordinary users rather than by AI researchers.
  • Because caption-based reasoning beats direct image input for the best settings, progress on FLIP may come as much from better text-to-story reasoning as from better vision encoders.
  • A 15-model ensemble reaches 85.2%, above the 71% threshold used to incentivize an open-sourced solver, suggesting that combining weakly correlated models is the most effective known strategy on this benchmark.
  • Small interventions that usually help reasoning—task reframing and historical exemplars—do not close the gap; reframing gives only a 2.0-point mean gain and exemplar contexts of size 2 to 16 reduce accuracy for the two strongest tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's contamination defense is indirect—it shows one model knows about the task concept, not that the evaluated models have not memorized the exact images or caption patterns; a direct membership test on the released dataset would settle whether FLIP measures reasoning or recall.
  • The consistent advantage of captions over raw pixels suggests that current models detect story coherence better from symbolic event descriptions than from scenes; a natural extension is to build a variant where captions are automatically paraphrased each time, forcing the reasoner to re-derive the story.
  • Since model errors are only weakly correlated and ensembles gain steadily, FLIP could be used as a testbed for diversity-aware model combination, where the goal is not one stronger model but a committee whose members fail on different flips.
  • A testable prediction following from the paper's framing is that models trained on order-ranking objectives over captioned image sequences—without ever seeing original FLIP images—should transfer to the benchmark, and measuring that transfer would separate reasoning skill from visual memorization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FLIP, a benchmark of 11,674 human-generated visual story-ordering tasks sourced from the Idena blockchain, split into train, validation, and test sets. The authors evaluate a broad range of open- and closed-sourced models, including vision-language models and captioning-plus-LLM pipelines, and report maximum accuracies of 75.5% for open-sourced and 77.9% for closed-sourced models in zero-shot settings, against 95.3% human accuracy, with a 15-model ensemble reaching 85.2%. The central claim is that current AI systems are far from human-level multimodal sequential reasoning, and that FLIP is a valid instrument for measuring this gap.

Significance. If the reported numbers are confirmed on a properly held-out test split, FLIP could be a valuable benchmark: it uses consensus-validated human ground truth from a blockchain, has a simple two-alternative design that facilitates error analysis, and the paper provides a broad comparison across many models, captioners, and prompting strategies. The caption-over-raw-images result and the observation that model predictions are only weakly correlated are interesting and potentially useful for future ensemble work. However, the headline accuracies are currently not backed by a held-out evaluation, and the selection over many model and configuration combinations makes the specific percentages unreliable; the benchmark's validity as a measure of reasoning is therefore not yet established.

major comments (4)
  1. [§3.1/Table 1, §4.1/Table 6, §3.4.4/Table 9] The manuscript never states which data split was used for the general benchmark and subsequent main experiments; internal evidence indicates that Tables 6-9 report validation-set performance. In particular, Section 3.4.4 says the historical-context experiment performs "inference on the validation data," and Table 9 states the context-window-0 column "corresponds exactly to the setting in the general benchmark, so the numbers in that column are from Table 6." The abstract and introduction present the resulting maxima (75.5%, 77.9%, 85.2%) as benchmark results, yet no result is reported on the 4,670-flip test split or the 106-flip test-short split defined in Table 1. Because the central claim is the human-AI gap on the FLIP benchmark, the scores need to be reported on the held-out test split; as written, the central quantitative claim is not established.
  2. [§4.1 and §C.2 (sample sizes and uncertainty)] The paper does not report the sample size behind any accuracy in Tables 6-9. If the general benchmark was run on validation-short (105 flips), the standard error of each accuracy is about 4.8 percentage points, so differences of a few points are within noise; moreover, the "maximum accuracies" are selected over the many model-by-caption combinations in Tables 6 and 13, and the 95.3% human figure is computed over all 84,600 participant answers rather than over the same evaluation set. The paper should state the evaluation split and sample size for every result, and provide confidence intervals or significance tests for the headline comparisons.
  3. [§C.3/Table 14 (ensemble evaluation)] The ensemble result of 85.2% is obtained by selecting the best among 1,000 random subsets of size 15 fitted on the (validation) data, and the "best subset of size 3/5" rows similarly search over all subsets on the same set. Selecting the best model class on the evaluation set yields optimistically biased accuracy. A properly held-out test split or nested cross-validation procedure is required before "ensembles achieve >85% accuracy" can be reported as a result.
  4. [§3.3.5 (data contamination)] The contamination argument is not tested. The paper only demonstrates that ChatGPT knows about FLIP challenges; it does not check whether the specific test images or caption patterns appear in the training corpora of the evaluated models. Given that the benchmark's public availability is acknowledged in Section 3.3.5, the authors should provide evidence (e.g., overlap analysis between the test flips/captions and model training data, or results on a fresh set of flips) that the reported accuracies measure reasoning rather than memorization.
minor comments (5)
  1. [Abstract and §4.1/Table 8] The abstract says "even the best open-sourced and closed-sourced models achieve maximum accuracies of 75.5% and 77.9%, respectively, in zero-shot settings," but the 75.5% appears in Table 8 (task reframing) while 77.9% is from Table 6 (general benchmark); please clarify that these come from different prompt settings, or use a single setting for the headline numbers.
  2. [§4 (introduction to results)] The text says "a simple ensemble of 5 open-sourced models can get an accuracy of 80.1%," but Table 14 shows this is the best subset of size 5 selected after exhaustive search, not a simple majority vote of five arbitrary models; the wording is misleading.
  3. [§3.2 vs. Table 10 caption] Section 3.2 defines the minimum performance criterion as accuracy above 71%, while the caption of Table 10 says "minimum performance of a successful model of 70%"; make these consistent.
  4. [References] Some reference entries contain malformed URLs with a missing space after "URL" (e.g., Hendrycks et al. 2021); please fix the formatting throughout the bibliography.
  5. [Figure 5] The correlation matrix in Figure 5 has no axis labels or colorbar legend; adding them would help readers identify which model pairs correspond to the high-correlation entries discussed in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: FLIP labels come from external human consensus and the reported model accuracies are zero-shot evaluations, not fits to the benchmark labels.

full rationale

The central claim is an empirical evaluation claim. FLIP ground-truth labels are determined by Idena consensus voting before any model is run, and the model accuracies in Tables 6 and 13 are zero-shot inference results with no parameter fitting on the test labels. The human 95.3% figure is a dataset statistic computed from the same consensus votes that define ground truth, but this is the standard annotation-consensus construction of a benchmark and does not enter the model scoring as a fitted parameter or as a predicted quantity derived from the models. The paper's self-citations (Plesner et al. 2024 for reCAPTCHAv2 and Pascual et al. 2021 for story generation) are motivational or related-work references and are not load-bearing; no uniqueness theorem and no ansatz is imported from prior author work to force a modeling choice. The ensemble result involving 'best of 1k subsets' is a selection/multiple-comparisons concern, not a circular derivation, and the possible validation-split ambiguity raised by the skeptic is a measurement-validity concern, not a self-referential reduction. No quoted equation or protocol makes a reported prediction equal to its own input by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's central claim is empirical and rests on the reliability of blockchain consensus labels, the representativeness of the scraped unencrypted flips, and the assumption that contamination via public Idena data is mitigated by captioning. No mathematical derivation is present, so free parameters are limited to evaluation hyperparameters and ensemble selection. No new physical or theoretical entities are introduced.

free parameters (4)
  • k-NN neighbor count k = 18
    Chosen for the ResNet50 embedding baseline; reported best k-NN accuracy is 58.9% (Appendix A).
  • MLP hidden layer sizes = 2000 and 100 units
    Selected for the baseline MLP; reported accuracy is 62.3% (Appendix A).
  • Ensemble subset size and selection budget = 3, 5, 15; best of all subsets of size 3/5 and best of 1,000 random subsets of size 15
    Table 14 reports the maximum over many subsets, which is a multiple-comparison selection on the test set rather than an unbiased estimator.
  • Context window sizes for historical context = 2, 4, 8, 16
    Chosen for the exemplar experiment in Section 4.4 and Table 9.
assumptions (3)
  • domain assumption Idena consensus votes provide reliable ground truth labels for which ordering tells a coherent story.
    Section 3.1 uses blockchain consensus scores as labels and discards no-consensus flips; no independent human relabeling is reported.
  • domain assumption Excluding encrypted epochs and no-consensus flips leaves a representative, unbiased sample of FLIP challenges.
    Section 3.1 collects only epochs 6 to 36 and removes all no-consensus flips; representativeness is asserted, not tested.
  • ad hoc to paper Public exposure of FLIP images on the Idena website does not meaningfully contaminate model evaluations when captions are used.
    Section 3.3.5 argues caption restatement reduces contamination, but provides no empirical test and shows ChatGPT already knows about FLIP tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLIP Reasoning Challenge." pith.science (2026). https://pith.science/paper/IALPBXRR

@misc{pith2026250412256,
  author       = {Pith},
  title        = {Pith review of: FLIP Reasoning Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IALPBXRR}},
  note         = {Machine review of arXiv:2504.12256}
}
read the original abstract

Over the past years, advances in artificial intelligence (AI) have demonstrated how AI can solve many perception and generation tasks, such as image classification and text writing, yet reasoning remains a challenge. This paper introduces the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks on the Idena blockchain. FLIP challenges present users with two orderings of 4 images, requiring them to identify the logically coherent one. By emphasizing sequential reasoning, visual storytelling, and common sense, FLIP provides a unique testbed for multimodal AI systems. Our experiments evaluate state-of-the-art models, leveraging both vision-language models (VLMs) and large language models (LLMs). Results reveal that even the best open-sourced and closed-sourced models achieve maximum accuracies of 75.5% and 77.9%, respectively, in zero-shot settings, compared to human performance of 95.3%. Captioning models aid reasoning models by providing text descriptions of images, yielding better results than when using the raw images directly, 69.6% vs. 75.2% for Gemini 1.5 Pro. Combining the predictions from 15 models in an ensemble increases the accuracy to 85.2%. These findings highlight the limitations of existing reasoning models and the need for robust multimodal benchmarks like FLIP. The full codebase and dataset will be available at https://github.com/aplesner/FLIP-Reasoning-Challenge.

Figures

Figures reproduced from arXiv: 2504.12256 by the authors.

Figure 1
Figure 1. Example of a Flip chal￾lenge from the Idena blockchain. The user is given 4 images pre￾sented in two different orderings (also referred to as stacks), and the user must select which stack of images tells a meaningful story. In this example, the answer is the right stack with the story of taking flour, mixing it with other ingre￾dients, frying the dough, and then getting pancakes. Since this is a coherent story, then… view at source ↗
Figure 3
Figure 3. Number of words in the captions provided by [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Distribution of how many flips a certain number of models correctly labels. For instance, [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Correlation between model predictions for all 33 models we consider. The first 27 are the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Examples of two flip challenges where all the best open-source models fail to predict the [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 12 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abdin, M. et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219

    arXiv 2024

  3. [3]

    Alayrac, J.B. et al. Flamingo: a visual language model for few-shot learning, 2022. URL https://arxiv.org/abs/2204.14198

    arXiv 2022

  4. [4]

    and Choi, Y

    Bisk, Y., Zellers, R., Bras, R.L., Gao, J. and Choi, Y. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    arXiv 2019

  5. [5]

    Bommasani, R. et al. On the opportunities and risks of foundation models, 2022. URL https://arxiv.org/abs/2108.07258

    arXiv 2022

  6. [6]

    Brown, T.B. et al. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165

    arXiv 2020

  7. [7]

    and Lee, Y.J

    Cai, M., Liu, H., Mustikovela, S.K., Meyer, G.P., Chai, Y., Park, D. and Lee, Y.J. Making large multimodal models understand arbitrary visual prompts, 2023

    work page 2023

  8. [8]

    and Bansal, M

    Cho, J., Lei, J., Tan, H. and Bansal, M. Unifying vision-and-language tasks via text generation, 2021. URL https://arxiv.org/abs/2102.02779

    arXiv 2021

Show all 55 references
  1. [9]

    On the measure of intelligence, 2019

    Chollet, F. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547

    2019 arXiv

  2. [10]

    and Tafjord, O

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C. and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    2018 arXiv

  3. [11]

    Cobbe, K. et al. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  4. [12]

    and Cucchiara, R

    Cornia, M., Stefanini, M., Baraldi, L. and Cucchiara, R. Meshed-memory transformer for image captioning, 2020. URL https://arxiv.org/abs/1912.08226

    2020 arXiv

  5. [13]

    Driess, D. et al. Palm-e: An embodied multimodal language model, 2023. URL https://arxiv.org/abs/2303.03378

    2023 arXiv

  6. [14]

    Glazer, E. et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024. URL https://arxiv.org/abs/2411.04872

    2024 arXiv

  7. [15]

    Grattafiori, A. et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  8. [16]

    and Yang, M

    Guan, X., Zhang, L.L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F. and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking, 2025. URL https://arxiv.org/abs/2501.04519

    2025 arXiv

  9. [17]

    and Sun, J

    He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

    2015 arXiv

  10. [18]

    and Steinhardt, J

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D. and Steinhardt, J. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  11. [19]

    Idena Blockchain explorer , 2025 a

    Idena . Idena Blockchain explorer , 2025 a . URL https://scan.idena.io/. Accessed: 2025-01-08

    2025

  12. [20]

    What is a flip? , 2025 b

    Idena . What is a flip? , 2025 b . URL https://www.idena.io/flip-challenge. Accessed: 2025-01-09

    2025

  13. [21]

    Idena Whitepaper , 2025 c

    Idena . Idena Whitepaper , 2025 c . URL https://docs.idena.io/docs/wp/technology. Accessed: 2025-01-10

    2025

  14. [22]

    and Qiu, J

    Jocher, G., Chaurasia, A. and Qiu, J. Ultralytics yolov8, 2023. URL https://github.com/ultralytics/ultralytics

    2023

  15. [23]

    and Girshick, R

    Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C.L. and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URL https://arxiv.org/abs/1612.06890

    2016 arXiv

  16. [24]

    and Sabharwal, A

    Khot, T., Khashabi, D., Richardson, K., Clark, P. and Sabharwal, A. Text modular networks: Learning to decompose tasks in the language of existing models, 2021. URL https://arxiv.org/abs/2009.00751

    2021 arXiv

  17. [25]

    and Morgenstern, L

    Kocijan, V., Davis, E., Lukasiewicz, T., Marcus, G. and Morgenstern, L. The defeat of the winograd schema challenge, 2023. URL https://arxiv.org/abs/2201.02387

    2023 arXiv

  18. [26]

    and Iwasawa, Y

    Kojima, T., Gu, S.S., Reid, M., Matsuo, Y. and Iwasawa, Y. Large language models are zero-shot reasoners, 2023. URL https://arxiv.org/abs/2205.11916

    2023 arXiv

  19. [27]

    and Morgenstern, L

    Levesque, H., Davis, E. and Morgenstern, L. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012

    2012

  20. [28]

    and Hoi, S

    Li, J., Li, D., Savarese, S. and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. URL https://arxiv.org/abs/2301.12597

    2023 arXiv

  21. [29]

    and Lee, Y.J

    Liu, H., Li, C., Li, Y. and Lee, Y.J. Improved baselines with visual instruction tuning, 2023

    2023

  22. [30]

    Liu, L. et al. Grin: Gradient-informed moe, 2024. URL https://arxiv.org/abs/2409.12136

    2024 arXiv

  23. [32]

    Nye, M. et al. Show your work: Scratchpads for intermediate computation with language models, 2021. URL https://arxiv.org/abs/2112.00114

    2021 arXiv

  24. [33]

    Chatgpt: Language model by openai, 2023

    OpenAI. Chatgpt: Language model by openai, 2023. URL https://chat.openai.com/. Accessed: 2024-06-01

    2023

  25. [34]

    Gpt-4 technical report, 2024

    OpenAI et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  26. [35]

    and Wattenhofer, R

    Pascual, D., Egressy, B., Meister, C., Cotterell, R. and Wattenhofer, R. A plug-and-play method for controlled text generation. arXiv preprint arXiv:2109.09707, 2021

    2021 arXiv

  27. [36]

    and Wattenhofer, R

    Plesner, A., Vontobel, T. and Wattenhofer, R. Breaking reCAPTCHAv2 . In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pp.\ 1047--1056. IEEE, 2024

    2024

  28. [37]

    and Yang, M

    Qi, Z., Ma, M., Xu, J., Zhang, L.L., Yang, F. and Yang, M. Mutual reasoning makes smaller llms stronger problem-solvers, 2024. URL https://arxiv.org/abs/2408.06195

    2024 arXiv

  29. [38]

    Qwen2.5 technical report, 2025

    Qwen et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  30. [39]

    Radford, A. et al. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    2021 arXiv

  31. [40]

    and Choi, Y

    Sap, M., Rashkin, H., Chen, D., LeBras, R. and Choi, Y. Socialiqa: Commonsense reasoning about social interactions, 2019. URL https://arxiv.org/abs/1904.09728

    2019 arXiv

  32. [41]

    and Kohli, P

    Saxton, D., Grefenstette, E., Hill, F. and Kohli, P. Analysing mathematical reasoning abilities of neural models, 2019. URL https://arxiv.org/abs/1904.01557

    2019 arXiv

  33. [42]

    and Soricut, R

    Sharma, P., Ding, N., Goodman, S. and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 2556--2...

    2018

  34. [43]

    and Kumar, A

    Snell, C., Lee, J., Xu, K. and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314

    2024 arXiv

  35. [44]

    and Artzi, Y

    Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H. and Artzi, Y. A corpus for reasoning about natural language grounded in photographs, 2019. URL https://arxiv.org/abs/1811.00491

    2019 arXiv

  36. [45]

    and Berant, J

    Talmor, A., Herzig, J., Lourie, N. and Berant, J. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv.org/abs/1811.00937

    2019 arXiv

  37. [46]

    Team, G. et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  38. [47]

    Wang, P. et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  39. [48]

    and Dong, Y

    Wang, Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O. and Dong, Y. Helpsteer2-preference: Complementing ratings with preferences, 2024 b . URL https://arxiv.org/abs/2410.01257

    2024 arXiv

  40. [49]

    and Le, Q.V

    Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M. and Le, Q.V. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  41. [50]

    and Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  42. [51]

    Yue, X. et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL https://arxiv.org/abs/2311.16502

    2024 arXiv

  43. [52]

    and Choi, Y

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  44. [53]

    Zhou, D. et al. Least-to-most prompting enables complex reasoning in large language models, 2023. URL https://arxiv.org/abs/2205.10625

    2023 arXiv

  45. [54]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  46. [55]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  47. [56]

    Zhao and Kelvin Guu and Adams Wei Yu and Brian Lester and Nan Du and Andrew M

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.