{"id":"e70026d5-db42-4a29-8fa4-0a213b451b9c","arxiv_id":"2505.05040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a balanced, manually translated set of Latvian food tweets, Llama 3.2 11B significantly outperforms the previous LLaVA baseline for image-text relation classification, while most other recent open-source VLMs do not.","lead":"This paper tests five open-source vision-language models on a new balanced set of 350 Latvian food tweets with manual English translations, asking whether images add to the tweet text and whether the text appears in the image. It finds that the newest models, such as Llama 3.2 11B, beat an older baseline, but most models still perform close to chance on this task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undisclosed output parsing and the Llama-only format prompt make the reported significant advantage unverifiable; the central claim needs a reproducibility check.","rationale":"The paper's central claim is empirical and modest: among five vision-language models on a balanced LTEC subset, Llama 3.2 11B is the only model whose improvement over the prior LLaVA 1.5 baseline is statistically significant. For that claim to hold, the reported accuracies must come from a fair, model-independent mapping of raw generations to the two binary questions and then to the four-class label. The paper never specifies this mapping, and it reveals in Section 4.3 that outputs could be 'gibberish word salad, repetitions, empty strings or otherwise unquantifiable outputs.' This is a routine reproducibility gap, not an accusation of bad faith. Because only Llama received the explicit '**Answer:** YES/NO' format instruction, its outputs are much easier to parse than those of the other models, which means the comparison is not controlled for prompt format or for parser compatibility. The reader's weakest assumption pointed to the unstated parsing step and the unequal prompt format, and I agree that this is the most load-bearing issue. The proposed test — releasing raw outputs and applying one fixed parser to all models — would settle whether the reported significance survives. Since the issue is addressable and the rest of the paper is transparent, I keep the reader's conditional verdict rather than moving to acceptance, rejection, or unverdictable.","tokens_in":9911,"tokens_out":6393,"duration_ms":68559,"concrete_test":"Release the raw generations for the 350-tweet balanced set across the 10 seeds for all five models, together with the exact parser used to convert text into YES/NO and then into the four-class label. Then apply a single pre-registered parser, treating unparseable outputs as errors for every model, and recompute the Table 3 accuracies; if the Llama 3.2 11B versus LLaVA 1.5 baseline gap no longer remains statistically significant under this uniform parser, the reported result depends on the undisclosed parsing step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that Llama 3.2 11B significantly outperforms the previous baseline on the balanced LTEC subset — rests on the accuracies in Table 3. Those accuracies depend on converting free-form model text into the two binary answers (Q1 and Q2) and then into the four-class label, but the paper never describes this conversion. Section 4.3 reports that Latvian-prompt runs produced 'gibberish word salad, repetitions, empty strings or otherwise unquantifiable outputs' instead of expected YES/NO answers, yet no parsing rule is given, and no statement says whether such outputs are treated as errors, discarded, or coerced by string matching. The same section reveals a second confound: only Llama 3.2 received the output template '**Answer:** YES/NO; **EXPLANATION:** ...', so its generations are likely to be cleanly parseable, while the other models received the older prompt and may require lossy extraction. If unparseable outputs were counted differently across models, or if the parser was tuned to Llama's format, the 33.07 vs 25.71 class-accuracy gap and the claim that 'only the result from Llama 3.2 11B is statistically significant' could be artifacts of the parsing protocol rather than of model capability. The zero variance of Qwen2-VL (15.71 ± 0.00) further suggests a degenerate output-to-label mapping for at least one model, and this is not explained.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five open-source vision-language models (Llama 3.2 11B, LLaVA-NeXT 7B/13B, Qwen2-VL 7B, Phi 3.5 4B) on the four-class image-text relation prediction task for tweets. The authors construct a balanced 350-tweet evaluation subset from the Latvian Twitter Eater Corpus (LTEC), obtain professional manual English translations, and also sample 450 tweets from the TIRT dataset. Across 10 random seeds, Llama 3.2 11B achieves the highest class accuracy on the balanced LTEC subset (33.07 on Latvian text, 33.83 on English text), and the paper states that only this result is statistically significant compared with the Rikters et al. (2024) LLaVA 1.5 7B baseline (25.71). Additional findings concern language sensitivity and domain sensitivity across models. The paper also includes a small machine-translation quality comparison of four MT systems against the manual translation.","tokens_in":10179,"tokens_out":3973,"duration_ms":39635,"significance":"If the results are supported, the paper contributes a useful balanced evaluation resource and a manual English translation of a low-resource multimodal dataset, and it provides a falsifiable statement about the current capabilities of open VLMs on this task. The authors are honest about statistical significance in the main table and run 10 seeds for the central comparison. However, the central quantitative claim currently rests on an undocumented output-parsing protocol and on a prompt that was modified only for the top-performing model; until these are resolved, the magnitude of the reported improvement cannot be verified. The dataset and translation effort are still valuable regardless of the resolution of the parsing question.","major_comments":[{"comment":"The output parsing step is never specified. The text reports that Latvian-prompt runs produced 'gibberish word salad, repetitions, empty strings or otherwise unquantifiable outputs' instead of expected YES/NO answers, but no rule is given for converting free-form model text into the two binary answers (Q1 and Q2) or for combining those answers into the four-class label. In addition, only Llama 3.2 received the appended output-format instruction ('Format the answer in the pattern of **Answer:** YES/NO; **EXPLANATION:** ...'), which makes its outputs much easier to parse automatically. As a result, the reported gap between Llama 3.2 (33.07/33.83) and the baseline (25.71) could be an artifact of the parsing protocol and prompt asymmetry rather than of model capability. Please state the exact parsing procedure, specify how non-compliant outputs are treated (e.g., counted as errors, discarded, or coerced by string matching), and verify the main comparison under a uniform output-format prompt for all models.","section":"§4.3, Table 3"},{"comment":"The baseline comparison is not clearly controlled. The baseline row is described as 'the highest scoring run from Rikters et al. (2024) using the LLaVA 1.5 model with 7B parameters', but the earlier work evaluated on the full 812-tweet set with automatic translations, whereas the current models are evaluated on the 350-tweet balanced subset with manual translations. If the baseline numbers were not recomputed on the identical balanced 350-tweet set, with the same manual English translations, the same seeds, and the same prompting protocol, then the comparison in Table 3 mixes differences in data, translation, and evaluation procedure with differences in model capability. Please clarify whether the baseline was re-run under the current protocol, and if not, supply a re-evaluation.","section":"§4.2, §5, Table 3"},{"comment":"The TIRT results are reported as single-run accuracies with no variance, no confidence intervals, and no significance testing, and the sampling procedure for the 450-tweet subset is not described in terms of seeds or sampling strategy. The claim that Llama 3.2 11B is the 'worst overall performer' on TIRT and the related domain-sensitivity conclusion rest entirely on this table, so as presented the differences could be within sampling noise. Please report multiple runs or otherwise quantify uncertainty for the TIRT evaluation.","section":"§5, Table 4"},{"comment":"The exact zero standard deviation (15.71 ± 0.00 for class accuracy on both Latvian and English text, and 47.71/47.43 for Q1) across 10 different random seeds is unexplained. This strongly suggests either deterministic decoding or that the model produced identical (likely non-compliant) outputs on every run. Because this bears directly on the output-parsing question, the paper should state the sampling parameters used and give examples of what Qwen2-VL actually generated.","section":"Table 3, Qwen2-VL rows"}],"minor_comments":[{"comment":"The emoji/category symbols used to denote the four classes do not render in the text; the class labels should be spelled out explicitly in the table and in the surrounding discussion.","section":"Table 1"},{"comment":"The machine-translation quality comparison (Table 2) motivates the manual translation but is not used further in the evaluation; a sentence clarifying its role would help the reader understand why it is included.","section":"§4.2"},{"comment":"The paper states that the balanced evaluation set and evaluation code will be released, but no repository link, dataset license, or code availability statement is provided; adding these would materially support the reproducibility claim.","section":"§6, Limitations"},{"comment":"There are several typographical inconsistencies, including the 'LLaV A' spacing in the model names and the unusual 'Preo¸ tiuc-Pietro' character rendering; a careful proofreading pass is needed before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's main empirical claim is plausible but is currently under-specified in the two places that matter most: the output-parsing protocol and the comparability of the baseline. I do not see signs of intentional misreporting, but the combination of a model-specific prompt and an undisclosed parser makes the headline number unverifiable from the manuscript alone. The balanced dataset and manual translation are a genuine contribution, and the paper is within scope for a natural-language-processing venue. The issues are fixable with additional reporting and a small amount of re-analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about zero-shot VLM evaluation on low-resource social-media text, but the headline comparison is not yet verifiable. What's genuinely new: a balanced 350-tweet subset of the Latvian Twitter Eater Corpus, professional English translations, and a five-model open-source VLM comparison with 10 seeds on LTEC plus a TIRT sample. Credit where due: the authors noticed prior evaluation was skewed by class imbalance, did the balancing properly, used a human translator instead of MT, reported Q1/Q2 separately, and candidly stated that only Llama 3.2's result is statistically significant. They also re-ran the old baseline on their balanced set, which is the right move.\n\nThe soft spots are real and mostly in the evaluation pipeline. Section 4.3 says Latvian-prompt runs produced 'gibberish word salad, repetitions, empty strings or otherwise unquantifiable outputs,' but the paper never states how free-form text was parsed into YES/NO or how the two binary answers were combined into the four-class label. That matters because the main claim—Llama 3.2 at ~33% vs. baseline ~25.7%—depends entirely on those conversions. The prompt confound is explicit: only Llama 3.2 got the 'Format the answer in the pattern...' instruction, so its outputs are likely cleaner and easier to parse. If unparseable outputs were counted as errors for other models and coerced for Llama, the gap could be an artifact. Qwen2-VL's zero variance across 10 seeds is a red flag that its outputs map degenerately to labels. The TIRT numbers have no error bars or significance tests, and the dropped Latvian-prompt condition narrows the language-robustness conclusions.\n\nThe paper is honest about its own limits; it does not overclaim in the text. But the central comparison cannot be checked without the parsing recipe and the balanced set, which are not yet released. The manual translations are also taken as ground truth without a quality check, though that is minor given they were done by a professional.\n\nBottom line: this is a solid candidate for a serious referee, not a desk reject. The balanced benchmark and the comparative snapshot are useful to people working on multilingual multimodal social media. But the authors should be asked to specify the parsing rule, run all models with the identical prompt format, and report variance on TIRT. I would want to see that revision before relying on the Llama 3.2 result.","headline":"Small honest empirical paper whose useful balanced benchmark is undercut by an undocumented output-parsing step and a prompt template only the winning model received.","tokens_in":10726,"tokens_out":2464,"would_cite":false,"duration_ms":25807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a balanced Latvian-English tweet benchmark, the open-source Llama 3.2 11B vision-language model reaches 33.8 percent four-class accuracy, beating the previous LLaVA baseline of 25.7 percent with statistical significance.","keywords":["image-text relation","vision-language models","multilingual tweets","Latvian language","zero-shot classification","balanced benchmark","social media","low-resource language"],"falsifier":"Re-run the five models on the same 350 balanced tweets with the published prompt and a written, deterministic rule for converting raw outputs into YES/NO answers, then combine the two answers into the four-class label exactly as the rule specifies; the central claim holds only if Llama 3.2 11B still beats the 25.71% baseline with statistical significance under that rule. A concrete adversarial check is to map empty or unparseable outputs to a default class and see whether the ranking in Table 3 survives.","tokens_in":9700,"feed_emoji":"🖼️","tokens_out":8569,"duration_ms":70405,"temperature":0.7,"pith_summary":"The paper claims that recent open-source vision-language models are becoming genuinely better at deciding how a tweet's image relates to its text, including in a low-resource language such as Latvian. To test this, the authors built a class-balanced 350-tweet evaluation set from the Latvian Twitter Eater Corpus, commissioned a manual English translation of the tweets, and probed five recent small-to-medium open models with yes/no questions. The central result is that Llama 3.2 11B reaches 33.07% four-class accuracy on Latvian text and 33.83% on the English translation, significantly beating the 25.71% LLaVA 1.5 7B baseline, while most other current models stay near chance. The paper further shows that model rankings reverse across domains, so no single open model is reliable everywhere yet.","feed_headline":"Llama 3.2 vision beats old baseline on tweet image-text relations","feed_subtitle":"On a balanced Latvian-English tweet set it hits 33.8% accuracy, yet most rival models stay near chance.","key_machinery":"The machinery is the decomposition of the four image-text relation categories into two independent yes/no questions, Q1 (does the image add to the text meaning?) and Q2 (is the text represented in the image?), which lets a vision-language model answer with simple YES/NO tokens that are then combined into the final four-class label. Around that lies the evaluation apparatus: a class-balanced 350-tweet subset of the Latvian Twitter Eater Corpus, a professional manual translation into English, a fixed zero-shot prompt with a special format instruction added for Llama 3.2, and ten random seeds per model-condition to stabilise the estimate. The same two-question protocol is run on a 450-tweet sample of the English TIRT corpus to expose domain sensitivity.","core_discovery":"On the paper's own terms, the discovery is that a 2024-generation open vision-language model, Llama 3.2 11B, outperforms the previous state of the art at image-text relation prediction on a balanced, manually translated tweet benchmark, and does so with statistical significance: 33.07 ± 0.36 four-class accuracy with the original Latvian text and 33.83 ± 0.17 with the English translation, against 25.71 ± 4.00 for the LLaVA 1.5 7B baseline. The paper also establishes that the other tested models (LLaVA-NeXT 7B and 13B, Phi 3.5 4B, Qwen2-VL 7B) do not reliably beat the baseline, and that performance is highly sensitive to both input language and data domain, with Llama 3.2 11B being the best on the Latvian food-tweet set but the worst on the English TIRT sample. The authors conclude that the task remains far from solved even though newer checkpoints are steadily improving.","pith_inferences":["The undocumented parsing step is my main reservation: until the authors describe or release it, the precise margins in Tables 3 and 4 should be read as provisional, since a parser that defaults empty or 'gibberish' outputs to one particular answer could inflate or deflate individual models.","Qwen2-VL's 15.71% score with zero variance across seeds suggests it may have collapsed into a near-fixed answer pattern on the Latvian set; inspecting its raw YES/NO distribution would settle whether it is guessing.","The paper itself leaves in-context learning and fine-tuning for future work; a small fine-tune of Llama 3.2 11B on a few hundred examples would directly test whether the 33.8% ceiling is a model limit or a prompting limit."],"forward_implications":["If the result is right, a 2024-generation open vision-language model can beat the previous state of the art on low-resource image-text relation prediction, and later checkpoints are likely to keep raising this ceiling.","The released balanced evaluation set and manual translations give other researchers a fairer, reproducible benchmark for multilingual tweet image-text relations than the skewed original split.","Prompt format is load-bearing: Llama 3.2 needed an added format instruction to produce clean answers, and asking the models to respond in Latvian produced unusable output, so prompt design must be reported alongside any deployment in a low-resource language.","Domain sensitivity is a practical obstacle: because Llama 3.2 ranks first on the Latvian set and last on the TIRT sample, a single benchmark cannot certify a model's general capability on this task."],"supporting_citations":[{"why":"Supplies the LTEC dataset, its four-class annotations, the original LLaVA 1.5 baseline numbers, and the prompt template the paper extends.","marker":"Rikters et al. (2024)"},{"why":"Introduces the four image-text relation categories and the TIRT dataset used for the domain-sensitivity comparison.","marker":"Vempala and Preotiuc-Pietro (2019)"},{"why":"Releases the LLaVA models, including the 1.5 7B checkpoint whose reported scores serve as the baseline.","marker":"Liu et al. (2023)"},{"why":"Releases Llama 3.2 Vision, the model that achieves the paper's best results and drives the central claim.","marker":"Dubey et al. (2024)"},{"why":"Releases LLaVA-NeXT Vicuna checkpoints (7B and 13B) evaluated in the comparison.","marker":"Li et al. (2024)"},{"why":"Releases Phi 3.5 Vision, one of the five evaluated models.","marker":"Abdin et al. (2024)"},{"why":"Releases Qwen2-VL, the fifth evaluated model and the lowest scorer on the Latvian set.","marker":"Bai et al. (2023)"}],"fun_headline_variants":["Llama 3.2 vision beats older baseline on tweet relations","New open VLM tops multilingual tweet benchmark, still low","Language and domain shift results for image-text tweet models","Llama 3.2 11B leads tweet image-text task, task unsolved","Tweet relation prediction: Llama 3.2 up, others near chance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies depend on a conversion of each model's free-form answer (which the paper admits sometimes came out as 'gibberish word salad' or empty strings) into a clean YES/NO label, and this parsing step is never described, so the numbers in the result tables could in principle be shaped by the parser rather than by the models.","fun_headline_variants_meta":{"raw":{"variants":["Llama 3.2 vision beats older baseline on tweet relations","New open VLM tops multilingual tweet benchmark, still low","Language and domain shift results for image-text tweet models","Llama 3.2 11B leads tweet image-text task, task unsolved","Tweet relation prediction: Llama 3.2 up, others near chance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1311,"prompt_tokens":866,"completion_tokens":445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":482,"tokens_out":445,"duration_ms":4652,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:14:37.217459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five models on the same 350 balanced tweets with the published prompt and a written, deterministic rule for converting raw outputs into YES/NO answers, then combine the two answers into the four-class label exactly as the rule specifies; the central claim holds only if Llama 3.2 11B still beats the 25.71% baseline with statistical significance under that rule. A concrete adversarial check is to map empty or unparseable outputs to a default class and see whether the ranking in Table 3 survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LTEC dataset, its four-class annotations, the original LLaVA 1.5 baseline numbers, and the prompt template the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the four image-text relation categories and the TIRT dataset used for the domain-sensitivity comparison."}],"review_version":1}