{"id":"97dd7a68-0fb1-4a6a-b6c2-9c245af2c5fe","arxiv_id":"2411.18968","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Machine-generated captions are more similar to human labels than object-detection labels, and ML captions give the best region-classification performance while combined ML objects and captions give the best income prediction.","lead":"This study compares how humans and machine models describe images of daily life from different countries and income levels. It finds that machine-generated captions are often as good as or better than human labels for some tasks, but human input is still needed for others.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The income-regression 'best overall' claim is not backed by a relevant significance test: the reported KS test compares the best model to the worst one, while the gap to Human Labels alone (25 RMSE points) is likely not significant.","rationale":"Good-faith reading: the paper contributes a new crowdsourced Dollar Street annotation corpus and a careful descriptive comparison; RQ1 results (ML Captions and Human Labels most similar) are internally consistent with the t-SNE and length statistics, and the dataset is released. The main risk is in RQ2's predictive claims. The strongest claim explicitly names the income RMSE winner, but the reported significance testing does not actually compare that winner to the human-only or all-annotation models. Given the means and SDs in Table 4, the difference to Human Labels is about 1.4% and likely within noise. Because the abstract and conclusion build the 'not replaceable' narrative on this and on the action vs non-action pattern, the statistical gap is load-bearing. The reader's weakest assumption (non-independence of the 10 runs) is a valid concern, but the more direct problem is the choice of comparator: a KS test of best vs worst cannot establish that the best is better than the relevant alternatives. I therefore recommend keeping the CONDITIONAL verdict, with the condition expanded to require a proper paired comparison of the income models.","tokens_in":17369,"tokens_out":8760,"duration_ms":79536,"concrete_test":"Re-run the income regression for all seven annotation combinations using the same 10 random seeds and the same 80/20 split, so the runs are paired; then compute paired t-tests and bootstrap 95% CIs for the RMSE difference between ML Obj+ML Capt and (a) Human Labels, and (b) All-annotations. Additionally, repeat the whole comparison on 100 random 80/20 splits to obtain a distribution of rank orderings. If the CI includes zero or the best-model rank is not stable across splits, the 'best overall' income claim should be downgraded to 'not distinguishable from human labels.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim for RQ2 has two parts. For region classification, ML Captions (F1 0.41) is compared with Human Labels (0.35); a Welch test is reported and this difference is plausibly real. For income regression, the claim that 'ML Objects and ML Captions performed best overall' rests on Table 4: best RMSE 1817.49 ± 37.13 vs Human Labels 1842.57 ± 14.99 and All-annotations 1842.89 ± 30.83. The paper's only significance test for income is a Kolmogorov-Smirnov test between this best model and the worst model, ML Captions & Human Labels (2006.36 ± 58.58), not against the nearest human-inclusive competitors. On the reported 10-run means/SDs, an unpaired Welch test of best vs Human Labels gives p ≈ 0.07, and best vs All-annotations p ≈ 0.11; these are not significant at conventional levels. Thus the headline 'performed best overall' for income regression is not statistically supported, and the conclusion that human annotations are not replaceable is correspondingly weakened. The reader's independence concern is real, but even granting independence this comparator problem remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares human-generated annotations (MTurk labels) with machine-generated annotations (Faster R-CNN object labels and BLIP captions) on 1,886 Dollar Street images spanning eight everyday-life categories. RQ1 measures annotation similarity via fine-tuned Sentence Transformer embeddings, cosine similarity, t-SNE, and annotation lengths. RQ2 trains a RUSBoost classifier to predict geographical region and an AdaBoost.R2 regressor to predict income from the annotation embeddings, using single and combined annotation sets. The main reported findings are that ML Captions and Human Labels are the most similar annotation pair, that ML Captions give the best overall region classification F1 (0.41), that ML Objects plus ML Captions give the lowest income RMSE (1817.49), and that human input is more effective for non-action image categories. The authors conclude that human annotations remain important and are not yet replaceable by machine annotations.","tokens_in":17603,"tokens_out":5428,"duration_ms":48110,"significance":"If the results were fully supported, the paper would provide a useful empirical datapoint on the substitutability of human and machine annotations in a socio-economically diverse image setting, and it would offer a new crowdsourced annotation dataset. Strengths include the release of code and data, the use of a real-world dataset with regional and income diversity, and the coverage of multiple annotation types and combinations. The main limitations are statistical: the income-regression 'best overall' claim is not backed by a significance test against the nearest competitors, the reported tests assume independence across runs that share a single test split, and the predictive tasks lack random or constant baselines. In addition, the text embedding model is fine-tuned on the very annotation texts that are later used for similarity and prediction, which complicates interpretation. These issues are fixable, so the contribution is defensible after revision.","major_comments":[{"comment":"The claim that 'ML Objects and ML Captions performed best overall' for income regression is not statistically supported. The only significance test reported for this task is a Kolmogorov-Smirnov test comparing the best model (ML Objects & ML Captions, RMSE 1817.49 ± 37.13) with the worst model (ML Captions & Human Labels, RMSE 2006.36 ± 58.58). This does not test the difference that matters for the headline claim, namely the gap to the nearest human-inclusive competitors: Human Labels alone (1842.57 ± 14.99) and all three annotations (1842.89 ± 30.83). Using the reported 10-run means and standard deviations, an unpaired Welch t-test of best versus Human Labels gives p ≈ 0.07, and best versus all-annotations gives p ≈ 0.11; neither is significant at the conventional 0.05 level. Please re-run the significance analysis against all pairwise competitors (or report simultaneous confidence intervals) and adjust the claim accordingly.","section":"Experiment 2"},{"comment":"The significance tests in Experiment 2 assume that the 10 runs with different random seeds are independent samples. However, all runs use the same 80/20 train/test split, so the F1 and RMSE values are correlated across runs. Welch's t-test and the Kolmogorov-Smirnov test are therefore anti-conservative, and the reported p-values (1.15e-7 and p < 0.001) overstate the reliability of the performance differences. A paired test across seeds, or better, multiple independent test splits, should be used. At minimum, the correlation should be acknowledged and the conclusions stated with appropriate caveats.","section":"Experiment 2"},{"comment":"The predictive experiments lack baselines. The region classification task has four roughly balanced classes, so a random classifier would achieve about 0.25 F1; the best reported overall F1 is 0.41, which is only modestly above chance. More importantly, some per-category scores fall below chance: for example, the Front Doors category with Human Labels reports F1 = 0.18 in Table 3. Without a random or majority-class baseline, the reader cannot interpret such values. For income regression, the RMSE of a constant predictor (e.g., predicting the mean income in the training set) should be reported as a reference point, because RMSE values around 1800–2000 are meaningful only relative to such a baseline. Please add these baselines and discuss them.","section":"Experiment 2"},{"comment":"The Sentence Transformer model ('all-MiniLM-L12-v2') is fine-tuned on the three annotation sets before the embeddings are computed. These same embeddings are then used both for the RQ1 similarity analysis and as features for the RQ2 prediction tasks. This is a self-referential step: the representation used for measuring similarity and for prediction has been adapted to the very texts being compared and predicted. Furthermore, because the fine-tuning uses all annotation texts, it includes texts that are later placed in the test split for RQ2, which is a form of leakage at the feature-extractor level. The authors should justify this design choice, compare with off-the-shelf embeddings, and/or fine-tune only on the training split within a cross-validation loop.","section":"Data Preparation and Preprocessing"},{"comment":"The conclusion that 'human-generated annotations are yet to be replaceable' is stronger than what the experiments show. The paper tests whether annotation embeddings are predictive of region and income, not whether human annotations can be replaced in a general annotation pipeline. The results may show complementarity in these two proxy tasks (e.g., human labels help for some non-action categories), but that is not the same as non-replaceability. I recommend softening the claim to something like 'human and machine annotations are complementary in the studied tasks' or explicitly defining what 'replaceable' would mean operationally.","section":"Abstract and Conclusion"}],"minor_comments":[{"comment":"The enumerated image categories incorrectly list 'washing clothes' twice and omit 'Drying Clothes'; the earlier dataset description should be used to fix this list.","section":"Data Annotation"},{"comment":"The text states that similarity scores are 'generated by the BLIP model's cosine similarity function,' but the methodology clearly uses the Sentence Transformer for embeddings and cosine similarity; this appears to be an error and should be corrected.","section":"Results and Discussion, RQ1"},{"comment":"In the description of Figure 5, please clarify what the line of best fit is fitted against when the x-axis is on a log scale and the y-axis is linear, and state whether the RMSE values in Table 4 are computed on the original or log-transformed income scale.","section":"Results and Discussion, RQ2"},{"comment":"The text refers to 'the ML Captions-Human Model' when describing the worst-performing regression model; this should be 'ML Captions & Human Labels Model' for consistency with Table 4.","section":"Results and Discussion, RQ2"},{"comment":"The Limitations paragraph contains only a single sentence about generalizability. It should also mention the statistical concerns (shared test split, missing baselines, leakage from fine-tuning) so the reader is aware of these caveats without having to infer them from the experimental details.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The reader's and skeptic's concerns are confirmed on reading the full manuscript: the income-regression headline is not backed by an appropriate significance test, and the shared test split plus fine-tuning on all annotations are additional load-bearing issues. The paper is not fatally flawed; the central experimental setup is reasonable and the data release is a positive contribution. However, the statistical validation must be redone and baselines added before the claims can be accepted. The conclusion also overreaches slightly relative to the evidence, but that can be fixed with rewording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate empirical study with a new crowdsourced dataset and a clean comparison of three annotation types on Dollar Street. The descriptive similarity results (RQ1) are plausible, and the authors are honest about scope. The problem is that the headline claim for RQ2 income regression is not statistically supported as written: the KS test compares best vs worst, not best vs nearest human-inclusive competitor. That needs fixing before the \"ML annotations performed best overall\" claim can stand.\n\nWhat's new: 1,886 Dollar Street images with three crowd labels per image, plus BLIP captions and Faster R-CNN object labels. Releasing code and data on GitHub is a real plus. The finding that ML captions cluster with human labels (cosine 0.69) while object labels are more distinct is a useful descriptive result, and the region-consistency analysis (Europe clustering) is interesting. The per-category breakdown for classification is a nice touch. The authors also correctly note the narrow scope in limitations.\n\nSoft spots: On the classification side, the Welch test between ML Captions (0.41) and Human Labels (0.35) is the right comparison, though the 10 runs share the same test split, so the p-value overstates reliability, and there are no random or majority-class baselines. On the regression side, the stress-test note is correct: the KS test compares the best model (ML Obj & ML Capt, RMSE 1817) to the worst (ML Capt & Human, 2006), not to Human Labels (1842) or All-annotations (1842). On the reported means/SDs, best vs Human Labels gives p≈0.07 and best vs All gives p≈0.11—not significant. So \"performed best overall\" is not supported. Also, the sentence transformer is fine-tuned on the same annotation texts used for both similarity and prediction; that's a self-referential step that should be acknowledged or ablated. The image filtering (only images where ML objects at ≥50% confidence and ML captions exist) could bias the sample toward easy-to-caption images; the authors mention it but don't analyze its effect.\n\nWho it's for: people working on annotation practice, algorithmic bias, and human-AI comparison. It deserves a serious referee: the dataset and question are worth engaging, but the statistical claims need revision. I would send it to review and ask for corrected significance tests and baselines.","headline":"Solid empirical comparison of human vs machine annotations on Dollar Street, but the headline income-regression claim rests on a significance test against the wrong baseline.","tokens_in":18118,"tokens_out":2422,"would_cite":true,"duration_ms":21452,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On 1,886 Dollar Street images, machine captions beat human labels at region classification, but human annotations still win where context is subtle.","keywords":["human annotation","machine-generated annotation","foundation models","image captioning","object detection","annotation similarity","algorithmic bias","Dollar Street dataset"],"falsifier":"Re-run both predictive experiments with fresh random splits of the data each time rather than the same 80/20 split, and recompute the significance tests; if the differences between best and worst annotation models no longer reach significance, the central reliability claim fails.","tokens_in":17171,"feed_emoji":"🖼️","tokens_out":6310,"duration_ms":54394,"temperature":0.7,"pith_summary":"This paper asks whether machine-generated image annotations can replace human-written ones, and it answers with a qualified no. Comparing crowdsourced human labels, object-detection labels, and image captions on 1,886 images from the Dollar Street dataset, the paper finds that machine captions and human labels are the most similar pair of annotations, while object labels are more distant. When the annotations are used to train predictive models, machine captions alone achieve the best overall region-classification score, and object labels combined with captions achieve the lowest income-prediction error. Yet human labels remain more effective for non-action categories such as living rooms and places for dinner, so the paper concludes that machine annotations are useful but not yet a full replacement for human ones.","feed_headline":"Machine captions beat human labels at region guessing","feed_subtitle":"On 1,886 Dollar Street images, machine captions win at region guessing yet human labels lead where context is subtle.","key_machinery":"The central mechanism is a comparison-and-prediction pipeline built on sentence embeddings. All three annotation sets—object labels from Faster R-CNN with a ResNet-50-FPN backbone, captions from BLIP with a Vision Transformer backbone, and crowdsourced human labels—are embedded by a fine-tuned Sentence Transformer model into 384-dimensional vectors; cosine similarity between these vectors measures annotation similarity, and the same vectors train a RUSBoost classifier for region prediction and an AdaBoost.R2 regressor for income prediction.","core_discovery":"The paper's central claim is that human and machine annotations of the same images overlap in vocabulary and sentence structure but differ in what they emphasize, and that neither source is uniformly better for downstream prediction. At the similarity level, ML Captions and Human Labels show the highest mean cosine similarity (0.69), while ML Objects sits lower (0.49 to 0.50); all three annotation types show consistent cross-region patterns, with European images receiving the strongest agreement. At the prediction level, ML Captions yield the best overall region-classification F1 score (0.41), and ML Objects combined with ML Captions yield the best overall income-regression RMSE (1817.49). The paper reports these gaps as statistically significant and interprets the category-level results as evidence that ML annotations work best for action categories, human labels work best for non-action categories, and combining annotation types often gives more consistent predictions.","pith_inferences":["A testable extension: retrain the captioning and detection models on geographically balanced training data and check whether the European agreement advantage disappears, which would point to training-data geography rather than universal perception.","Because every run reuses the same 80/20 test split, the reported significance levels treat correlated samples as independent; re-running with fresh splits would likely widen the error bars and produce more conservative p-values.","The category-level pattern suggests a practical division of labor: use machine annotations for scalable coverage and add human labels selectively for categories where ML error is systematically higher.","The categories where humans win are exactly those requiring inference about material conditions, so an automated system that switches annotation strategy by category could approximate human quality at lower cost."],"forward_implications":["Region-classification models trained on ML captions alone can outperform those trained on human labels or object labels, suggesting captions carry the most region-discriminative signal.","For income prediction, combining object labels with captions gives the lowest overall error, but no single annotation type dominates every region or image category.","ML annotations are strongest for action-based categories, while human labels remain stronger for static or context-dependent categories such as living rooms and places for dinner.","Human and machine annotations show consistent cross-region patterns, so replacing human labels with machine ones does not by itself remove geographic bias.","Combining annotation types often yields more consistent predictions, such as the lower variance seen for African samples, than relying on a single source."],"supporting_citations":[{"why":"Supplies the Dollar Street images with region and income labels that all three annotation sets describe.","marker":"Gapminder 2021"},{"why":"Defines the Faster R-CNN object detection model used to generate the ML Objects annotations.","marker":"Ren et al. 2017"},{"why":"Defines the BLIP captioning model used to generate the ML Captions annotations.","marker":"Li et al. 2022"},{"why":"Provides the COCO dataset that trained the object detection model.","marker":"Lin et al. 2014"},{"why":"Provides the COCO Captions dataset that trained the BLIP captioning model.","marker":"Chen et al. 2015"},{"why":"Supplies the pre-trained Sentence Transformer model fine-tuned to embed all annotations.","marker":"Reimers 2021"},{"why":"Defines the RUSBoost classifier used for the region classification experiment.","marker":"Seiffert et al. 2008"},{"why":"Defines the AdaBoost.R2 algorithm used for the income regression experiment.","marker":"Drucker 1997"}],"fun_headline_variants":["AI captions beat humans at region guessing, but not all tasks","Human labels still top for non-action image details","ML captions win region classification; humans keep non-action edge","Image annotations: AI and humans overlap in style, differ in focus","Neither AI nor human annotations fully replaceable, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability claims assume that the 10 runs with different random seeds are independent samples, even though every run reuses the same test images, so the reported significance levels likely overstate how much the results would generalize.","fun_headline_variants_meta":{"raw":{"variants":["AI captions beat humans at region guessing, but not all tasks","Human labels still top for non-action image details","ML captions win region classification; humans keep non-action edge","Image annotations: AI and humans overlap in style, differ in focus","Neither AI nor human annotations fully replaceable, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3251,"prompt_tokens":918,"completion_tokens":2333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2231}},"tokens_in":534,"tokens_out":2333,"duration_ms":15551,"temperature":1.0,"reasoning_tokens":2231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:40:50.050050+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both predictive experiments with fresh random splits of the data each time rather than the same 80/20 split, and recompute the significance tests; if the differences between best and worst annotation models no longer reach significance, the central reliability claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Dollar Street images with region and income labels that all three annotation sets describe."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Faster R-CNN object detection model used to generate the ML Objects annotations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained Sentence Transformer model fine-tuned to embed all annotations."},{"cited_title":"M.; Van Hulse, J.; and Napolitano, A","cited_arxiv_id":null,"evidence_quote":"Defines the RUSBoost classifier used for the region classification experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the AdaBoost.R2 algorithm used for the income regression experiment."}],"review_version":1}