{"id":"5781d6e1-eb34-40fd-90fb-48dcfbbb4beb","arxiv_id":"2504.12256","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The FLIP benchmark of 11,674 blockchain image-story puzzles shows best open and closed AI models reach 75.5% and 77.9% accuracy, below the 95.3% human consensus baseline.","lead":"This paper introduces FLIP, a benchmark of 11,674 image-ordering puzzles from the Idena blockchain, and tests whether modern AI models can pick the logically coherent story. The best models reach about 75 to 78 percent accuracy versus about 95 percent for humans, and even a 15-model ensemble only reaches 85 percent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracies appear to be measured on the validation split, not the held-out test split, so the central gap claim lacks a proper evaluation.","rationale":"The reader's conditional verdict is appropriate: the paper introduces a plausible benchmark, but its central claim is not yet fully supported. My stress-test identifies a more directly load-bearing issue than the reader's weakest-assumption pick (contamination): the main results table appears to be tied to validation data, not the held-out test split, by the paper's own cross-reference between Table 9 and Section 3.4.4. This is an internal consistency problem that can be checked and fixed, and it affects the validity of the exact numbers in the abstract. The contamination concern is real but secondary: even if contamination were ruled out, we would still need to know which split produced the reported scores. Because the issue is resolvable by reporting the correct split, I keep the verdict at CONDITIONAL/UNCHANGED rather than moving to REJECT. The reader's rationale already noted the under-specified test split, so there is partial agreement, but the reader's stated weakest assumption focuses on contamination rather than the split ambiguity.","tokens_in":16674,"tokens_out":5352,"duration_ms":55924,"concrete_test":"Inspect the released codebase and configuration files to identify which split is loaded for the general benchmark, or, if the code is not yet public, re-run the general-benchmark protocol (prompts in Table 3, same model versions) on the test-short split of 106 flips and, if feasible, on the full test split of 4670 flips. Report the maximum open-sourced and closed-sourced accuracies with 95% Clopper-Pearson confidence intervals. If the held-out test maxima differ materially from 75.5% and 77.9%, the headline claim must be revised and the benchmark re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—best open-sourced 75.5%, best closed-sourced 77.9%, human 95.3%—depends on the evaluation set being a proper held-out test split. The manuscript defines such a split (Table 1: test set of 4670 flips, test-short of 106 flips), but no main-results table states that this split was used. Evidence internal to the paper points the other way: Section 3.4.4 says the historical-context experiment performs 'inference on the validation data,' and Table 9 says its context-window-0 column 'corresponds exactly to the setting in the general benchmark, so the numbers in that column are from Table 6.' Thus Table 6, the source of the headline maxima, appears to report validation-set performance. If the validation split used is validation-short (105 flips), each accuracy is a binomial estimate with standard error around 5 percentage points, and taking the maximum across 70+ model/caption combinations (Tables 6 and 13) further inflates the reported figures. Without a confirmed held-out test evaluation, the measured human–AI gap is not established; the benchmark itself is not yet validated as an instrument. This is a correctness risk independent of data contamination, and it is prior to any contamination argument: we must know what split the scores came from before deciding whether they measure reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FLIP, a benchmark of 11,674 human-generated visual story-ordering tasks sourced from the Idena blockchain, split into train, validation, and test sets. The authors evaluate a broad range of open- and closed-sourced models, including vision-language models and captioning-plus-LLM pipelines, and report maximum accuracies of 75.5% for open-sourced and 77.9% for closed-sourced models in zero-shot settings, against 95.3% human accuracy, with a 15-model ensemble reaching 85.2%. The central claim is that current AI systems are far from human-level multimodal sequential reasoning, and that FLIP is a valid instrument for measuring this gap.","tokens_in":16943,"tokens_out":6986,"duration_ms":61296,"significance":"If the reported numbers are confirmed on a properly held-out test split, FLIP could be a valuable benchmark: it uses consensus-validated human ground truth from a blockchain, has a simple two-alternative design that facilitates error analysis, and the paper provides a broad comparison across many models, captioners, and prompting strategies. The caption-over-raw-images result and the observation that model predictions are only weakly correlated are interesting and potentially useful for future ensemble work. However, the headline accuracies are currently not backed by a held-out evaluation, and the selection over many model and configuration combinations makes the specific percentages unreliable; the benchmark's validity as a measure of reasoning is therefore not yet established.","major_comments":[{"comment":"The manuscript never states which data split was used for the general benchmark and subsequent main experiments; internal evidence indicates that Tables 6-9 report validation-set performance. In particular, Section 3.4.4 says the historical-context experiment performs \"inference on the validation data,\" and Table 9 states the context-window-0 column \"corresponds exactly to the setting in the general benchmark, so the numbers in that column are from Table 6.\" The abstract and introduction present the resulting maxima (75.5%, 77.9%, 85.2%) as benchmark results, yet no result is reported on the 4,670-flip test split or the 106-flip test-short split defined in Table 1. Because the central claim is the human-AI gap on the FLIP benchmark, the scores need to be reported on the held-out test split; as written, the central quantitative claim is not established.","section":"§3.1/Table 1, §4.1/Table 6, §3.4.4/Table 9"},{"comment":"The paper does not report the sample size behind any accuracy in Tables 6-9. If the general benchmark was run on validation-short (105 flips), the standard error of each accuracy is about 4.8 percentage points, so differences of a few points are within noise; moreover, the \"maximum accuracies\" are selected over the many model-by-caption combinations in Tables 6 and 13, and the 95.3% human figure is computed over all 84,600 participant answers rather than over the same evaluation set. The paper should state the evaluation split and sample size for every result, and provide confidence intervals or significance tests for the headline comparisons.","section":"§4.1 and §C.2 (sample sizes and uncertainty)"},{"comment":"The ensemble result of 85.2% is obtained by selecting the best among 1,000 random subsets of size 15 fitted on the (validation) data, and the \"best subset of size 3/5\" rows similarly search over all subsets on the same set. Selecting the best model class on the evaluation set yields optimistically biased accuracy. A properly held-out test split or nested cross-validation procedure is required before \"ensembles achieve >85% accuracy\" can be reported as a result.","section":"§C.3/Table 14 (ensemble evaluation)"},{"comment":"The contamination argument is not tested. The paper only demonstrates that ChatGPT knows about FLIP challenges; it does not check whether the specific test images or caption patterns appear in the training corpora of the evaluated models. Given that the benchmark's public availability is acknowledged in Section 3.3.5, the authors should provide evidence (e.g., overlap analysis between the test flips/captions and model training data, or results on a fresh set of flips) that the reported accuracies measure reasoning rather than memorization.","section":"§3.3.5 (data contamination)"}],"minor_comments":[{"comment":"The abstract says \"even the best open-sourced and closed-sourced models achieve maximum accuracies of 75.5% and 77.9%, respectively, in zero-shot settings,\" but the 75.5% appears in Table 8 (task reframing) while 77.9% is from Table 6 (general benchmark); please clarify that these come from different prompt settings, or use a single setting for the headline numbers.","section":"Abstract and §4.1/Table 8"},{"comment":"The text says \"a simple ensemble of 5 open-sourced models can get an accuracy of 80.1%,\" but Table 14 shows this is the best subset of size 5 selected after exhaustive search, not a simple majority vote of five arbitrary models; the wording is misleading.","section":"§4 (introduction to results)"},{"comment":"Section 3.2 defines the minimum performance criterion as accuracy above 71%, while the caption of Table 10 says \"minimum performance of a successful model of 70%\"; make these consistent.","section":"§3.2 vs. Table 10 caption"},{"comment":"Some reference entries contain malformed URLs with a missing space after \"URL\" (e.g., Hendrycks et al. 2021); please fix the formatting throughout the bibliography.","section":"References"},{"comment":"The correlation matrix in Figure 5 has no axis labels or colorbar legend; adding them would help readers identify which model pairs correspond to the high-correlation entries discussed in the text.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the stress-test note is right, and it matters. The paper defines a held-out test set (4,670 flips) but never reports evaluating on it. Section 3.4.4 says the historical-context experiment runs \"inference on the validation data,\" and Table 9's context-0 column is explicitly copied from Table 6, the source of the 77.9% closed-source headline. No main-results table states that the test split was used. So the central quantitative claim—75.5% open-source, 77.9% closed-source, 95.3% human—rests on validation performance, not a proper held-out evaluation. That is load-bearing, not a nit.\n\nWhat is new and good: the FLIP dataset itself is genuinely new to the literature, and the human-consensus baseline from the Idena blockchain gives it independent value. The comparison between reasoning from captions and reasoning from raw images is a clean, interesting finding, and the result that caption-based reasoning wins is worth knowing even if the absolute numbers shift. The ensemble analysis, showing that diverse models help, is also a useful data point.\n\nSoft spots beyond the split: the abstract's 75.5% open-source figure does not appear in the general benchmark table (where the best open-source number is 73.7%); it seems to come from the task-reframing table. That inconsistency is never explained. The short validation set is about 105 flips, so the percentages carry standard errors around 4–5 points, and the authors take maxima over 70+ model-caption combinations. The 85.2% ensemble result is the best of 1,000 random subsets fit on validation—a selected number, not a principled estimate. The contamination defense is asserted rather than tested: showing ChatGPT knows about FLIP tasks does not show models have not memorized the images or caption patterns. And the code and dataset are only promised, not shipped.\n\nTo be fair, the overall direction—models well below human consensus on these tasks—is plausible, and the ground truth comes from external human votes, so there is no circularity problem there. But as written, the specific numbers are not trustworthy, and the benchmark is not yet validated as an instrument.\n\nWho gets value: researchers working on multimodal reasoning benchmarks and anyone interested in proof-of-personhood CAPTCHAs. It deserves a serious referee because the dataset and question are real, but the revision needs a proper held-out evaluation, clear sample sizes or error bars, and code/data release before the numbers can be cited.","headline":"The FLIP dataset is a real addition to multimodal reasoning benchmarks, but the headline accuracies are measured on validation rather than held-out test, so the human-AI gap is not yet established.","tokens_in":17468,"tokens_out":5351,"would_cite":false,"duration_ms":49421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that FLIP, a four-image story-ordering benchmark built from human consensus, exposes a large gap: best models reach 77.9% while people reach 95.3%.","keywords":["FLIP benchmark","visual reasoning","sequential reasoning","visual storytelling","multimodal AI","image captioning","zero-shot evaluation","common-sense reasoning"],"falsifier":"Take a set of original FLIP challenges and an equal set of newly written flips that are structurally identical but have never been posted online; if models score much higher on the original set than on the new set, the reported gap is partly memorization rather than reasoning, while matching scores would support the paper's claim that FLIP measures genuine multimodal sequential reasoning.","tokens_in":16456,"feed_emoji":"🧩","tokens_out":11867,"duration_ms":111564,"temperature":0.7,"pith_summary":"The paper introduces FLIP, a benchmark made from thousands of human-verified visual tasks on a blockchain identity platform, where a solver sees the same four images in two different orders and must pick the order that tells a coherent story. Its central claim is that this seemingly easy task exposes a large gap in current AI: the best open-sourced model answers correctly 75.5% of the time and the best closed-sourced model 77.9%, while the people who generated the tasks reach 95.3%. The paper also reports that feeding models written captions of the images works better than showing them the images directly, and that combining 15 models raises accuracy to 85.2%. If the benchmark measures what it claims, then no current system reasons about event sequences at a human level, and FLIP provides a concrete instrument for tracking progress toward that ability.","feed_headline":"Best AI scores 77.9% on FLIP story-order benchmark; humans 95.3%","feed_subtitle":"Four-image story-order test separates machines from people; captions beat raw images, ensembles still trail","key_machinery":"The central object is the FLIP challenge: a pair of orderings, called stacks, of the same four images, where exactly one stack tells a meaningful story and the other does not; the correct answer comes from human consensus voting on the blockchain platform. The load-bearing mechanism in the evaluation is the caption pipeline: an image-captioning model (mainly the BLIP-2 family) turns each image into a text description, and a text-only reasoner then compares the two four-caption sequences and chooses the more coherent story. This separation lets the paper argue that contamination is unlikely, because captions restate each challenge in fresh text and the prompts never mention the source of the tasks.","core_discovery":"On the paper's own terms, the discovery is that FLIP challenges—four images presented in two alternative orderings, with ground truth set by human consensus voting—are easy for people but hard for every tested model in zero-shot settings. With no training examples, the strongest open-sourced reasoning model reaches 75.5% and the strongest closed-sourced model 77.9%, compared with a 95.3% human accuracy; a logistic-regression ensemble of 15 models reaches 85.2%. The paper also finds that converting images to text captions helps: one top model improves from 69.6% when reading four raw input images to 75.2% when reasoning over captions, and several captioning-plus-reasoning combinations cross the 71% threshold the paper defines for a successful open-sourced solver. Task reframing gives small average gains, while providing models with historical examples of their own past answers lowers accuracy, and an error analysis shows the best models still fail on many strong-consensus flips.","pith_inferences":["The paper's contamination defense is indirect—it shows one model knows about the task concept, not that the evaluated models have not memorized the exact images or caption patterns; a direct membership test on the released dataset would settle whether FLIP measures reasoning or recall.","The consistent advantage of captions over raw pixels suggests that current models detect story coherence better from symbolic event descriptions than from scenes; a natural extension is to build a variant where captions are automatically paraphrased each time, forcing the reasoner to re-derive the story.","Since model errors are only weakly correlated and ensembles gain steadily, FLIP could be used as a testbed for diversity-aware model combination, where the goal is not one stronger model but a committee whose members fail on different flips.","A testable prediction following from the paper's framing is that models trained on order-ranking objectives over captioned image sequences—without ever seeing original FLIP images—should transfer to the benchmark, and measuring that transfer would separate reasoning skill from visual memorization."],"forward_implications":["No tested model reaches the platform's human threshold of 92% accuracy, so FLIP currently separates machines from people on a task built by ordinary users rather than by AI researchers.","Because caption-based reasoning beats direct image input for the best settings, progress on FLIP may come as much from better text-to-story reasoning as from better vision encoders.","A 15-model ensemble reaches 85.2%, above the 71% threshold used to incentivize an open-sourced solver, suggesting that combining weakly correlated models is the most effective known strategy on this benchmark.","Small interventions that usually help reasoning—task reframing and historical exemplars—do not close the gap; reframing gives only a 2.0-point mean gain and exemplar contexts of size 2 to 16 reduce accuracy for the two strongest tested models."],"supporting_citations":[{"why":"Establishes that machines solve reCAPTCHAv2 at 100%, motivating a new benchmark that tests reasoning rather than recognition.","marker":"Plesner et al. (2024)"},{"why":"Documents how model performance drops on a standard reasoning benchmark when questions are restated, which is the contamination risk FLIP's captioning pipeline is designed to avoid.","marker":"Mirzadeh et al. (2024)"},{"why":"Defines the flip challenge and the task of choosing the order that tells a meaningful story, fixing the benchmark's task design.","marker":"Idena (2025b)"},{"why":"Supplies the verification thresholds and the observed human accuracy the paper uses as its success criteria and human baseline.","marker":"Idena (2025c)"},{"why":"Provides the BLIP-2 captioning models that produce the text descriptions used in the best-performing caption-plus-reasoning pipeline.","marker":"Li et al. (2023)"},{"why":"Underpins the historical-context experiment by showing that models can learn from written examples at inference time.","marker":"Wei et al. (2022)"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the publicly available FLIP challenges and the caption patterns have not been memorized by the tested models, so that solving them requires fresh reasoning rather than recall; the paper argues this is unlikely but does not directly test it against newly created flips.","fun_headline_variants_meta":{"error":"DeepSeek 429: {\"error\":{\"message\":\"Too many requests. Your current concurrency is 163, which exceeds your concurrency limit of 117 based on your remaining balance. Please top up your balance to restore your concurrency.\",\"type\":\"rate_limit_error\",\"param\":null,\"code\":\"invalid_request_error\"}}"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:08.030601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of original FLIP challenges and an equal set of newly written flips that are structurally identical but have never been posted online; if models score much higher on the original set than on the new set, the reported gap is partly memorization rather than reasoning, while matching scores would support the paper's claim that FLIP measures genuine multimodal sequential reasoning.","supporting_citations":[{"cited_title":"and Wattenhofer, R","cited_arxiv_id":null,"evidence_quote":"Establishes that machines solve reCAPTCHAv2 at 100%, motivating a new benchmark that tests reasoning rather than recognition."}],"review_version":1}