{"id":"dc8a28ba-b985-4da5-b327-0bb421ba9df1","arxiv_id":"2504.18912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multimodal LLMs, especially GPT-4o and Gemini, can infer plausible Stack Overflow questions from code and IDE screenshots with moderate similarity to the original posts, but performance drops on complex screenshots.","lead":"This paper tests whether three large language models (LLaMA, Gemini, GPT-4o) can look at screenshots of code or developer tools and automatically write the Stack Overflow question that goes with them. The best models, GPT-4o and Gemini, get a moderate match to the original human-written questions, but errors grow when screenshots are cluttered.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.59/0.69 headline mixes an uncalibrated embedding metric with relevance-to-screenshot (Q1); true alignment with the original question (Q2) tops out at 0.43.","rationale":"The reader's weakest assumption is the use of the original Stack Overflow question as the gold standard, and I agree that this is problematic. My stress-test sharpens the concern: the paper's own Table II shows Q2 human alignment at only 0.39-0.43, yet Section VII highlights the Q1 score of 0.69, which measures relevance to the screenshot rather than fidelity to the original question. The embedding-based numbers, including the 51.75% headline, are presented without a null baseline or any calibration, so they cannot be interpreted as evidence of successful question inference. This is a load-bearing weakness for the central claim, but it is addressable with a no-image control and a rephrased conclusion. It does not change the reader's CONDITIONAL verdict; it explains why that verdict is needed. I therefore recommend keeping the verdict unchanged, but with the explicit requirement that the authors add a no-image control or otherwise calibrate the embedding metric before the quantitative claims are taken at face value.","tokens_in":12353,"tokens_out":5034,"duration_ms":53918,"concrete_test":"Run the same inference pipeline with the image removed: prompt GPT-4o and Gemini with the identical in-context template but no screenshot (or a blank image), and compute the same all-MiniLM-L6-v2 similarities against the 143 original posts. If the no-image control produces mean similarity near 0.5 or a fraction above 0.6 close to 51.75%, the reported scores reflect generic programming-language priors rather than visual understanding of the screenshots, and the central claim would need substantial revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section VII rests on two numbers: embedding-based similarity up to 0.59 and developer-perceived relevance up to 0.69. The second number is Q1 from Table II, the relevance of the generated question to the screenshot, not Q2, the relevance of the generated question to the original Stack Overflow post. For Q2, GPT-4o's maximum is 0.43. The claim that models can 'infer' the posted question therefore leans on the 0.59 embedding similarity and the derived '51.75% of images above 60% similarity' statistic. Both are similarity-to-baseline measures that inherit the loose image-text alignment documented in Section VI-A, where the average relevance of the posted image to the original question (Q3) is only 0.69 and the Q3-Q2 correlation is 0.53. More importantly, the embedding score is uncalibrated: all-MiniLM-L6-v2 cosine similarity between generated text and a human-written post has no established interpretation as a measure of question-reconstruction quality. A generic Stack Overflow-style question could plausibly score near 0.5-0.6 against many programming posts, so '51.75% above 0.6' has no intrinsic meaning without a null baseline (e.g., generated with no image) or an upper bound (e.g., original question vs. itself). The paper reports no such controls and no significance tests, so the headline 'promising capabilities' for screenshot-to-question inference is, as stated, unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether vision-language LLMs can generate Stack Overflow questions from code or IDE screenshots, comparing Llama-3.2, Gemini-1.5-Pro, and GPT-4o under in-context learning, chain-of-thought, and few-shot prompting. The dataset consists of 143 Stack Overflow posts with a single code/IDE screenshot. The authors evaluate generated questions against the original post using all-MiniLM-L6-v2 embedding cosine similarity and developer ratings on three Likert questions: Q1 (relevance of the generated question to the screenshot), Q2 (relevance of the generated question to the original post), and Q3 (relevance of the original post to the screenshot). The reported headline results are embedding similarities up to 0.59 for GPT-4o and Gemini, a developer-perceived relevance of up to 0.69, and the claim that GPT-4o exceeds 60% similarity with the baseline for 51.75% of images. The paper concludes that state-of-the-art LLMs show promising capabilities for inferring questions from programming screenshots, while also acknowledging limitations around screenshot relevance, small sample size, and possible training-data exposure.","tokens_in":12531,"tokens_out":4278,"duration_ms":43470,"significance":"If the central claim were fully supported, the paper would provide useful early evidence on whether multimodal LLMs could enable screenshot-first technical Q&A, with practical implications for developer forums and debugging copilots. The study has notable strengths: it uses a manually filtered, recent dataset; it makes data and code available in a GitHub repository; the developer evaluation shows high inter-rater reliability (Cohen's kappa 0.86); and the authors are transparent about dataset size and the imperfect alignment between Stack Overflow images and their associated text posts. However, the paper's headline numbers as written conflate two different developer-rated measures (Q1 and Q2) and the embedding similarity is not calibrated against any null or upper bound, so the 'promising capabilities' conclusion outruns the evidence. The substantive results—particularly the Q2 alignment near 0.43 and the moderate Q3-Q2 correlation near 0.53—are more modest than the abstract and conclusion suggest.","major_comments":[{"comment":"The claim that 'perceived relevance scores from developers reach up to 0.69' is based on Q1 from Table II, which measures the relevance of the generated question to the screenshot, not Q2, which measures relevance to the original Stack Overflow question. The Q2 maximum is 0.43 (GPT-4o with in-context learning), and all Q2 values are in the 0.34–0.43 range. The abstract and Section VII therefore overstate what the developer evaluation supports; the statement that models can 'infer' the original posted question rests on the uncalibrated embedding similarity and the 51.75% statistic, not on the perceived-relevance measure cited. Please revise the abstract and conclusion to report Q1 and Q2 separately and to qualify any claim about reconstructing the original question.","section":"Abstract, Section VII"},{"comment":"The embedding-based similarity scores are presented without calibration. A cosine similarity of 0.59 between a generated question and the original post has no established interpretation as a measure of question-reconstruction quality, and the same is true for the derived '51.75% of images above 60% similarity' statistic. Without a null baseline (for example, questions generated by the same model with no image or with an unrelated image) and an upper bound (for example, the original question embedded against itself or a paraphrase oracle), the absolute value 0.59 cannot be read as 'promising'. The authors should either add such controls or reinterpret the scores relative to them, and they should state which all-MiniLM-L6-v2 similarities correspond to clearly relevant versus irrelevant programming questions.","section":"Section V-A, Table I"},{"comment":"The paper reports no confidence intervals, standard errors, or significance tests for any of the key comparisons. In Table I, the differences between GPT-4o and Gemini are often 0.01–0.02 (for example, 0.59 versus 0.58 in the Combined row), and in Table II the Q2 differences between models are also small (0.43 versus 0.41, 0.40, 0.39). With n=143 for the embedding evaluation and n=50 for the developer evaluation, these effect sizes are plausibly within sampling noise. The paper should report uncertainty estimates and, where relevant, significance tests or effect sizes before asserting that one model or one prompting technique is 'best' or that one is 'significantly' better than another.","section":"Sections V-A, V-B, VI"},{"comment":"The ground-truth assumption—that the original Stack Overflow post is the correct target that a model should produce from the screenshot—is load-bearing and is only partially addressed. Section VI-A reports Q3 (image relevance to the original post) averaged at 0.69 with a moderate Q3-Q2 correlation of 0.53, which means that for many posts the screenshot alone does not determine the original question. The conclusions in Section VII do not qualify the 'promising capabilities' statement accordingly. In addition, the two few-shot examples in Section IV-C (references [42] and [43]) are drawn from the same October 2024 dataset used for the evaluation, which contaminates the few-shot condition even though that condition underperformed; this should be disclosed as a threat to validity in Section VI.","section":"Sections IV-A, IV-C, VI-A"}],"minor_comments":[{"comment":"The sentence 'In particular, we answer:' introduces research questions that are never explicitly enumerated; either list the RQs or remove this dangling sentence.","section":"Section I"},{"comment":"The prompt figure is labeled 'Made a little short due to space constraint' and the text is visibly truncated; the full prompt templates should be included, either in the figure or in an appendix, for reproducibility.","section":"Figure 1"},{"comment":"The bulleted list of metrics uses '0:' and '�' as bullet symbols, which appear to be formatting artifacts; these should be replaced with standard bullet points or numbered list items.","section":"Section IV-B"},{"comment":"The asterisks attached to 0.59 and 0.68 are never explained in the captions or the text; please define the asterisk or remove it.","section":"Tables I and II"},{"comment":"The 'manual analysis' of the ten best- and worst-performing images does not report who performed the analysis, how the set was selected, or whether a second annotator was involved; this makes the qualitative finding hard to verify.","section":"Section VI-B"},{"comment":"There is a typo in the text: 'H UGGING FACE' should be 'Hugging Face'.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for MSR and the authors have made their artifacts publicly available, which is commendable. The main concern is that the headline conclusion is not aligned with the quantitative evidence as reported: the 0.69 developer relevance is Q1 (screenshot relevance), not Q2 (alignment with the original post), and the embedding metric needs calibration before 'over 60% similarity' can be interpreted. The lack of statistical inference further weakens comparisons between models. I would encourage the authors to resubmit after re-analysis and reframing, and I would also suggest they trim the reference list, which contains a large block of self-citations ([45]–[79]) of marginal relevance to the specific claims in this paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the task: generating a Stack Overflow question from a code or IDE screenshot, which the authors reasonably claim has no direct SE study in the cited literature. They built a dataset (though heavily filtered from 4,310 down to 143), ran three multimodal LLMs under three prompting strategies, and report both embedding similarity and developer ratings. Credit where due: the data and code are promised in a GitHub repo, the limitations section candidly admits the imperfect image-text alignment and small sample, and the inter-rater kappa of 0.86 is solid. This is an honest first benchmark, not a hype piece.\n\nBut the central claim in Section VII is misstated. The \"0.69\" perceived relevance is Q1, relevance of the generated question to the screenshot, not to the original posted question. For Q2, relevance to the original question, GPT-4o's best is 0.43. That is a meaningful difference, and the reader's stress-test note is correct to call it out. The embedding-based similarity numbers (0.59 peak, 51.75% above 0.6) also lack calibration. all-MiniLM-L6-v2 cosine similarity between a generated question and a human-written post has no intuitive threshold; without a null condition (e.g., generating a generic programming question with no image) or an upper bound (original vs. itself), the 0.59 values could mean a lot or very little. The paper reports no confidence intervals or significance tests either, so the model ordering is descriptive, not inferential.\n\nOther soft spots are more minor. The filtering pipeline from 4,310 to 143 posts could systematically select clearer screenshots; the few-shot examples come from the same dataset, which could advantage or disadvantage that condition; the final copilot vision goes beyond what the data support. None of this makes the paper worthless, but it does mean the phrase \"promising capabilities in interpreting screenshot-based programming queries\" should be read as \"models can produce relevant-looking text about a screenshot, but aligning with the actual user question remains weak.\"\n\nWho is this for? Researchers working on developer Q&A, debugging copilots, and multimodal SE. They will find the dataset and the prompt design useful as a starting point. It deserves a serious referee, but only with the requirement that the authors fix the Q1/Q2 reporting, add a calibration baseline for the embedding metric, and temper the conclusions to match the Q2 results.\n\nMy recommendation: send it to peer review, but expect heavy revision. The task novelty is enough to justify referee time, and the experimental skeleton is repairable.","headline":"A genuinely new benchmarking task with usable data, but the headline numbers overstate the models' ability to reconstruct the original Stack Overflow question: 0.69 is relevance to the screenshot, not to the original question (Q2 tops out at 0.43).","tokens_in":13155,"tokens_out":1698,"would_cite":false,"duration_ms":19259,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art vision-language models, especially GPT-4o and Gemini-1.5-Pro, can reconstruct plausible Stack Overflow questions from code or IDE screenshots alone for a substantial fraction of cases, with GPT-4o exceeding 60% similarity…","keywords":["large language models","Stack Overflow","screenshot-based queries","question generation","multimodal LLM","prompt engineering","code screenshot","IDE screenshot"],"falsifier":"Have human developers who see only the same screenshots write questions, then score LLM-generated questions against human-generated ones using the same embedding and Likert measures. If the humans substantially outperform GPT-4o and Gemini on the same 143 images, the 'promising capability' claim is not about screenshot understanding but about a lower bar. Conversely, if humans match the LLMs, the benchmark is image-limited, not model-limited.","tokens_in":12070,"feed_emoji":"🖼️","tokens_out":9536,"duration_ms":88214,"temperature":0.7,"pith_summary":"This paper asks whether a developer can skip writing a question and simply post a screenshot of code or an IDE, relying on an AI to turn the image into a proper Stack Overflow post. It tests three models (LLaMA-3.2, Gemini-1.5-Pro, GPT-4o) and three prompting strategies on 143 real Stack Overflow questions that include code or IDE screenshots, measuring how close each generated question comes to the original posted question. The paper's central finding is that GPT-4o and Gemini can reconstruct the original question's meaning for a large portion of clear screenshots, while LLaMA and few-shot prompting lag far behind. If the finding holds, image-centric bug reporting on developer forums becomes feasible with AI assistance, and debugging copilots that understand IDE screenshots are a realistic near-term goal.","feed_headline":"GPT-4o turns code screenshots into questions, 52% of the time","feed_subtitle":"The finding makes image-only bug reports with AI assistance a realistic near-term feature for developer forums.","key_machinery":"The central machinery is a benchmark that feeds each of 143 single-image Stack Overflow posts to an LLM through one of three prompt templates (in-context learning, chain-of-thought, and few-shot learning) and then compares the generated title and body with the original post. Comparison runs through two channels: cosine similarity between text embeddings produced by the all-MiniLM-L6-v2 sentence-embedding model, and developer Likert ratings for three questions (Q1 relevance of the generated question to the screenshot, Q2 relevance to the original Stack Overflow question, and Q3 relevance of the posted image to the posted question). This design separates model capability, prompting style, and image-content difficulty, and it is what lets the paper attribute performance gaps to the content of the screenshots.","core_discovery":"On its own terms, the paper claims that state-of-the-art multimodal LLMs, specifically Gemini and GPT-4o, show promising capability in interpreting screenshot-based programming queries. That capability is quantified two ways: embedding-based similarity between generated and original Stack Overflow text reached 0.59, and developers rated the relevance of generated questions to the screenshot at 0.67–0.68. The headline result is that GPT-4o generated questions with over 60% similarity to the original posted question for 51.75% of the 143 images. The paper also finds that the link between screenshot content and the posted question is imperfect, with average image relevance to the original question at 0.69, and that model-image alignment correlates only moderately with the screenshot's relevance to the original question. These numbers support the conclusion that screenshot-only questioning is feasible for simple, clear images but not yet reliable for complex or cluttered ones.","pith_inferences":["Beyond the paper, matching the original post may be the wrong success criterion for a screenshot-only feature; a generated question that is more accurate than the original text would be penalized by the embedding metric. An outcome-based evaluation, such as whether the generated question receives a correct and accepted answer, would be a stronger test of usefulness.","A testable extension suggested by the data is to pair each screenshot not with the original post but with a human-written question created by a developer who sees only the screenshot. If humans beat GPT-4o by a wide margin, the bottleneck is model vision; if humans do not, the benchmark is measuring image information content rather than model capability.","Because few-shot learning was consistently worst, the two manually selected examples may have steered generation toward those posts' style. Varying few-shot examples by language or framework would separate prompt-example effects from model limits.","The paper's results imply that an interactive assistant could classify screenshot complexity first and ask for a text clarification when the image is cluttered, reserving fully automatic conversion for high-confidence screenshots."],"forward_implications":["If correct, developer forums could offer an image-to-question assistant that turns a code or IDE screenshot into a formatted Stack Overflow-style post, lowering the barrier for users who struggle to articulate their problem.","Screenshot-only debugging copilots become a realistic near-term goal: the models already reconstruct the core problem from clear images, and further fine-tuning on programming screenshots should improve accuracy on complex cases.","The strong dependence on image content implies that practical tools should guide users to capture explicit error messages, code, and annotations rather than full cluttered IDE windows.","With temperature zero, in-context learning and chain-of-thought prompting give more consistent results than few-shot prompting, so deployment choices about prompting matter as much as model choice.","The moderate correlation between screenshot relevance and generated-question alignment suggests that improving screenshot-to-question reconstruction requires improving the input, not just the model."],"supporting_citations":[{"why":"Model card for Gemini-1.5-Pro, the multimodal model whose performance the paper measures.","marker":"[40]"},{"why":"Model card for Llama-3.2, the open-source baseline that trails behind the other models.","marker":"[41]"},{"why":"GPT-4o system card, the model that produced the best-performing generated questions.","marker":"[33]"},{"why":"Defines in-context learning, one of the three prompt strategies evaluated.","marker":"[30]"},{"why":"Introduces chain-of-thought prompting, the strategy that worked best for Gemini.","marker":"[27]"},{"why":"Source of few-shot learning, the strategy that performed worst across models.","marker":"[26]"},{"why":"Provides the all-MiniLM-L6-v2 embedding model used to compute semantic similarity.","marker":"[37]"},{"why":"Defines the cosine similarity metric used for the embedding-based scores.","marker":"[38]"},{"why":"Stack Exchange Data Explorer, the source from which the 143-question dataset was collected.","marker":"[36]"}],"fun_headline_variants":["GPT-4o reads code screenshots, writes questions half the time","Code screenshot to question: GPT-4o nails it 52% of the time","Half of code screenshots yield questions via GPT-4o","GPT-4o decodes screenshots into questions—but only half the time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the original Stack Overflow title plus body as the gold-standard target for a screenshot-only input, even though images and text are only loosely aligned in many posts (average relevance 0.69), so every similarity score measures agreement with an imperfect standard.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o reads code screenshots, writes questions half the time","Code screenshot to question: GPT-4o nails it 52% of the time","Half of code screenshots yield questions via GPT-4o","GPT-4o decodes screenshots into questions—but only half the time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2905,"prompt_tokens":899,"completion_tokens":2006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":1924}},"tokens_in":515,"tokens_out":2006,"duration_ms":13445,"temperature":1.0,"reasoning_tokens":1924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:05:52.679418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human developers who see only the same screenshots write questions, then score LLM-generated questions against human-generated ones using the same embedding and Likert measures. If the humans substantially outperform GPT-4o and Gemini on the same 143 images, the 'promising capability' claim is not about screenshot understanding but about a lower bar. Conversely, if humans match the LLMs, the benchmark is image-limited, not model-limited.","supporting_citations":[{"cited_title":"Semantic cosine simi - larity,","cited_arxiv_id":null,"evidence_quote":"Model card for Gemini-1.5-Pro, the multimodal model whose performance the paper measures."},{"cited_title":"Llama 3.2 model card,","cited_arxiv_id":null,"evidence_quote":"Model card for Llama-3.2, the open-source baseline that trails behind the other models."},{"cited_title":"Review of large vision models and visual prompt engineering,","cited_arxiv_id":null,"evidence_quote":"Defines in-context learning, one of the three prompt strategies evaluated."},{"cited_title":"Scaling up visual and vision -language representation learning with noisy text supervision,","cited_arxiv_id":null,"evidence_quote":"Source of few-shot learning, the strategy that performed worst across models."},{"cited_title":"model card","cited_arxiv_id":null,"evidence_quote":"Provides the all-MiniLM-L6-v2 embedding model used to compute semantic similarity."},{"cited_title":"Stack exchange data explorer","cited_arxiv_id":null,"evidence_quote":"Defines the cosine similarity metric used for the embedding-based scores."}],"review_version":1}