{"id":"231512c1-a1e6-496f-917b-b546fee21bda","arxiv_id":"2507.03330","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Recipe-derived object status phrases added to vision-language matching improve recipe-step prediction by 20 to 26 accuracy points on instructional and real-world non-visual cooking videos.","lead":"This paper presents OSCAR, a pipeline that guesses which recipe step a cook is on by matching video frames to short object-status phrases such as \"chopping carrots\" or \"sautéing mushrooms\". Across 173 instructional videos and 12 real cooking sessions filmed by blind and low-vision cooks, these status phrases improved step-prediction accuracy by about 20 to 26 percentage points over baseline image-text matching.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablation bundles object status with the time-causal filter; Tables 1–2 do not establish that object status, rather than monotonic temporal smoothing, drives the reported gains.","rationale":"The reader's CONDITIONAL verdict is already the right one, so this stress-test does not move the verdict. I agree with the reader's core concern that 'object status' is represented only as text phrases extracted from the recipe, with no visual object-status detector, and that the reported effect may therefore be a text-image retrieval artifact rather than object-status recognition. I would add a sharper internal-validity point: even granting that the status phrases are meaningful, Tables 1 and 2 do not isolate them from the time-causal model. Because the time-causal model is applied in the OSCAR condition but not the baseline, and because §4.4.2 explicitly says the time-causal model improved performance, the reported +20-point gains cannot be attributed to object status without an ablation. The proposed concrete test separates these factors: condition B isolates time-causal smoothing, condition C isolates object-status phrases, and condition E controls for phrase length and formatting. This single ablation would settle whether the title's central claim is supported. The paper's real-world dataset and qualitative design insights remain valuable regardless of the outcome, which is why conditional acceptance rather than rejection is appropriate. I mark agreement as partial because the reader's stated weakest assumption focuses on construct validity, whereas the most load-bearing hole is the missing attribution ablation; both point to the same need for revision, but they are distinct technical defects.","tokens_in":19925,"tokens_out":5341,"duration_ms":63649,"concrete_test":"Re-run the YouCook2 evaluation (at least the 173-video set; a pre-registered random subsample of 50 is acceptable if compute is limited) and the 12-session non-visual set with four conditions on the same sampled frames: (A) recipe-step text only (baseline); (B) recipe-step text + time-causal model; (C) object-status phrases only, no time-causal; (D) object-status + time-causal (full OSCAR). Compare B vs A and B vs D. If B — the time-causal treatment without any object-status text — recovers most of D's gain over A (e.g., at least 75% of the reported delta), the headline attribution to object status fails; if B stays near A and D retains the gain, the claim survives. As a format control, also run (E) recipe-step text rewritten by GPT-4o into short verb+noun phrases without status semantics; if E matches C, the effect is paraphrase length, not object-status content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — object status consistently improves step prediction accuracy (Abstract, §4.3.2, §5.3.2) — is not identifiable from the reported comparison. In §4.2.2 the OSCAR condition applies three changes at once: (1) similarity scores between frames and object-status phrases, (2) averaging of those scores with the recipe-step baseline scores, and (3) a time-causal model enforcing monotonic step order. The baseline condition applies none of these. Tables 1 and 2 therefore measure the combined effect, not the effect of object status. The paper's own qualitative analysis (§4.4.2) attributes part of the gain to the time-causal model, but no reported condition runs recipe-step similarity plus time-causal without object-status phrases, so the size of that part is unknown. A second, related gap is construct validity: the 'object statuses' are GPT-4o-extracted verb+noun phrases from the recipe text (§3.1), and OSCAR never detects object state changes in the video; it computes CLIP/SigLIP text-image similarity with those phrases. A shorter, differently worded paraphrase of the same step can improve retrieval accuracy without recognizing any object status. Thus the reported +20-point gains could be caused by temporal smoothing, by ensembling two text representations, or by phrase-format effects, rather than by object-status recognition. The missing ablation is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OSCAR, a pipeline for recipe-progress tracking that uses GPT-4o to reformat recipes and extract object-status phrases of the form [verb]+[noun] (Section 3.1), computes CLIP and SigLIP similarity between video frames and both the recipe-step text and the object-status phrases (Section 3.2), averages the two similarity scores, and applies a time-causal model that enforces monotonic step order (Section 3.4). The pipeline is evaluated on 173 YouCook2 videos and a newly collected dataset of 12 real-world kitchen sessions recorded by blind and low-vision cooks. The reported results show large accuracy gains over a recipe-step-only baseline: from 41.7% to 68.0% (CLIP) and 62.2% to 82.8% (SigLIP) on YouCook2, and from 33.7% to 58.4% and 41.9% to 66.7% on the non-visual cooking dataset (Tables 1 and 2). The paper also presents error analyses and design considerations around implicit tasks, lighting, camera placement, and pre-prepared ingredients.","tokens_in":20215,"tokens_out":4375,"duration_ms":52520,"significance":"If the central claim were supported, the paper would make a useful contribution to accessible cooking technology and to procedural understanding for vision-language models. The real-world dataset of 12 non-visual cooking sessions is a genuine and valuable resource, and the design considerations in Section 5.5 are grounded in concrete observations. The pipeline is clearly described and the qualitative error analysis is thoughtful. However, the headline assertion that 'object status consistently improves step prediction accuracy' is not identified by the reported experiments: the OSCAR condition changes several components at once, and the object-status phrases are generated from the same recipe text that defines the baseline, so the measured gains cannot be attributed to visual object-status recognition as currently framed. The paper's strengths are the dataset, the domain motivation, and the design insights; the technical attribution needs further experiments.","major_comments":[{"comment":"The central claim is not identifiable from the reported comparison. The OSCAR condition applies three changes simultaneously: (1) similarity against object-status phrases, (2) averaging of those scores with the recipe-step similarity scores, and (3) a time-causal model enforcing monotonic step order. The baseline applies none of these, so the 20+ point gains in Tables 1 and 2 measure the combined pipeline, not the effect of object status. The paper's own analysis attributes part of the improvement to the time-causal model (§4.4.2 for duplicate actions and §4.4.3, where 'adding the time-causal model corrected the prediction'), but no condition runs recipe-step similarity plus the time-causal model without object-status phrases, so the size of that contribution is unknown. The limitation section (§7) does not acknowledge this attribution gap. Please add at least four conditions — baseline, baseline+time-causal, baseline+object-status phrases without time-causal, and full OSCAR — and report per-video paired comparisons.","section":"§4.2.2, Tables 1–2, §4.4.2"},{"comment":"The construct validity of 'object status recognition' is not established. The object statuses are GPT-4o-extracted verb+noun phrases obtained from the ingredient list and cooking steps of the same recipe, and OSCAR never detects object state changes in the video; it computes CLIP/SigLIP text-image similarity between frames and these phrases. A shorter, differently worded paraphrase of the same step can improve retrieval accuracy without any recognition of object status, so the measured gain may reflect a reformatting or ensembling effect rather than the visual signal the title and RQ1 claim. A concrete test would be to compare the status phrases against control phrase sets (e.g., generic action phrases or paraphrases not derived from status semantics) and to evaluate whether the predicted statuses agree with manually annotated visual state changes.","section":"§3.1–§3.2"},{"comment":"The statistical support is insufficient for the claim that object status 'consistently improves' accuracy. No confidence intervals, paired significance tests, or per-video results are reported; the standard deviations are across videos, and the three repeated samples per step are not independent replicates. In particular, the assertion in §5.3.2 that 'These gains were consistent across all 12 videos' is unsupported without a per-video table, especially given the small n=12 dataset and the high variance reported. Please report per-video accuracies for all videos in both datasets and use a paired test (e.g., Wilcoxon signed-rank) to support the consistency claim.","section":"§4.3, §5.3, Tables 1–2"},{"comment":"The averaging of baseline and object-status similarity scores is underspecified. The text states that 'OSCAR then averaged the similarity metrics' (§3.2) and that 'the results were then averaged with the baseline similarity scores' (§4.2.2), but neither the weight nor the averaging formula is given, and no sensitivity analysis is provided. If equal weighting is intended, the magnitude of the reported gain depends on that untested hyperparameter; if another weighting was used, it should be stated. The blur-filter threshold and the exact GPT-4o prompt are also unspecified, which limits reproducibility of the pipeline.","section":"§3.2, §4.2.2"}],"minor_comments":[{"comment":"There is a typo: 'uoloaded' should be 'uploaded'.","section":"§4.5.4"},{"comment":"'Groundtruth' should be 'ground truth' for consistency with the rest of the paper.","section":"§4.2.1"},{"comment":"The model name is spelled inconsistently as both 'SigLIP' and 'SigLip'; please standardize.","section":"Throughout"},{"comment":"Reference [29], the CHI Extended Abstracts version, expands OSCAR as 'Object Status and Contextual Awareness for Recipes,' which differs from the acronym expansion used in this manuscript ('Object Status Context Awareness for Recipes'); please align the naming.","section":"References"},{"comment":"The claim that gains were consistent across all 12 videos would be much easier to verify if a supplementary table of per-video baseline and OSCAR accuracies were included.","section":"§5.3.2"},{"comment":"The description of ingredient extraction says information was sourced 'directly from the video content or the publisher's notes on YouTube'; please clarify whether this was done by the authors and how disagreements were resolved, since this affects reproducibility.","section":"§4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution and the qualitative design analysis are the strongest parts of the manuscript, and I expect the authors can address the technical attribution with additional experiments. The main risk is that the title and abstract overstate what is measured: the current experiments do not isolate object-status recognition from temporal smoothing and text reformatting. If the ablation results show that the time-causal model accounts for most of the gain, the paper would still be valuable as a pipeline and dataset contribution, but the central claim would need to be substantially reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper has a genuine contribution in its real-world non-visual cooking dataset, and its qualitative analysis of where vision-language models fail in that setting is worth reading. But the headline claim—that object status consistently improves step prediction—is not actually supported by the experiments as reported. The OSCAR condition bundles object-status similarity averaging with a time-causal filter, and the baseline includes neither. So the +20–26 point gains in Tables 1 and 2 could be driven by the temporal smoothing alone, or by ensembling two text representations, rather than by anything like recognizing object states.\n\nThe dataset is the strongest asset: 12 cooking sessions by blind and low-vision participants in their own kitchens, annotated with recipe steps, and slated for release. That is a real resource for accessibility AI. The paper also does well in cataloging failure modes—implicit preparatory tasks, tactile rechecking, pre-prepared ingredients lingering in view, poor lighting, off-center chest cameras—and translating them into design considerations. Those insights are useful regardless of the quantitative claim.\n\nThe soft spots are concentrated in the evaluation. There is no ablation that runs recipe-step similarity plus time-causal without object-status phrases. The paper even acknowledges in Section 4.4.2 that the time-causal model improved performance, but never reports that condition alone. Second, the \"object statuses\" are GPT-4o-extracted verb+noun phrases from the recipe text (e.g., \"chopping carrots\"); the pipeline never detects a state change in the video—it computes CLIP/SigLIP similarity between frames and these phrases. So the title overclaims what the system does. Third, there are no confidence intervals or significance tests for the accuracy differences, and no code or prompts are released yet. These are all fixable, but they make the central claim premature.\n\nWho gets value from this: accessibility and HCI researchers, and anyone prototyping assistive cooking systems. The dataset and the design considerations are worth engaging with. I would not cite the accuracy numbers as evidence for object-status recognition without the missing ablation.\n\nRecommendation: send to peer review, with a request for major revision. A serious referee should ask for the ablation separating object status from the time-causal model, a reframing of what \"recognition\" means here, and released artifacts. The paper has the bones of a solid contribution; the current evaluation just does not back the abstract's claim.","headline":"A useful dataset and a clear accessibility motivation, but the paper's central claim about object status is not isolated by the experiments—the gains could come from temporal smoothing or text ensembling.","tokens_in":20725,"tokens_out":1908,"would_cite":true,"duration_ms":25469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding object-status phrases to vision-language models lifts recipe-step prediction accuracy by 20–26 percentage points on both instructional videos and real non-visual cooking sessions.","keywords":["object status recognition","recipe progress tracking","non-visual cooking","vision-language models","CLIP","SigLIP","time-causal modeling","accessibility"],"falsifier":"Run the same 173-video evaluation with each object-status phrase replaced by a non-status paraphrase of equal length (for example, 'carrots in this step' in place of 'chopping carrots'); if accuracy stays at the OSCAR level, the gain is not caused by object status. Alternatively, take a set of frames in which the same ingredient is visibly whole and then visibly chopped and check whether CLIP/SigLIP similarity to the status phrase rises and falls with the actual visual state; if it does not, the pipeline is not recognizing object statuses.","tokens_in":19731,"feed_emoji":"🍳","tokens_out":8296,"duration_ms":79838,"temperature":0.7,"pith_summary":"This paper argues that the evolving condition of ingredients and tools—an onion being chopped, a sauce thickening—is a usable signal for deciding which recipe step a cook is on, and that adding this signal to vision-language models makes step prediction much more accurate. The authors build OSCAR, a pipeline that parses recipes, extracts object-status phrases as verb-plus-noun pairs such as 'chopping carrots,' scores video frames against both recipe steps and these phrases with CLIP or SigLIP, and applies a monotonic time-causal rule so predictions cannot jump backwards. On 173 YouCook2 instructional videos, per-step accuracy rises from 41.7% to 68.0% with CLIP and from 62.2% to 82.8% with SigLIP; on 12 home cooking sessions recorded by blind and low-vision participants, it rises from 33.7% to 58.4% and from 41.9% to 66.7%. A sympathetic reader would care because this is a step toward assistive cooking tools that know what stage of a recipe a user is at without requiring the user to hold a phone or follow a rigid script.","feed_headline":"Adding object-state text lifts recipe-step accuracy 20+ points","feed_subtitle":"On 185 cooking videos, matching frames to object-state phrases beats matching to full steps.","key_machinery":"The load-bearing mechanism is the object-status phrase itself: a [verb]+[noun] description such as 'chopping carrots' that is extracted from the recipe text and used as an additional text anchor for video frames. A vision-language encoder (CLIP or SigLIP) produces a similarity score between each frame and each recipe step, and a second similarity score between the same frame and each step's object-status phrase; the two scores are averaged, and a time-causal model then suppresses predictions of an earlier step once a later step has been reached. The object status is never detected in the image by a dedicated model; it enters the pipeline as text, so the work the phrase does is entirely through text-image similarity.","core_discovery":"The central claim is that object status recognition consistently improves recipe-step prediction accuracy across vision-language models, in both curated instructional videos and real-world non-visual cooking. OSCAR represents object status as verb-plus-noun phrases extracted from the recipe's ingredient list and steps, computes CLIP/SigLIP similarity between each video frame and both the full step text and these status phrases, averages the two similarity scores, and then enforces time-causal monotonicity on predicted step numbers. The paper reports that this combination lifts CLIP from 41.7% to 68.0% and SigLIP from 62.2% to 82.8% on YouCook2, and from 33.7% to 58.4% and 41.9% to 66.7% on its own 12-session non-visual cooking dataset, with gains in every one of the 12 real-world sessions. The authors interpret the mechanism as disambiguation: status phrases pull apart visually similar or cluttered frames, while the time-causal rule prevents duplicate or out-of-order steps from being predicted.","pith_inferences":["Beyond the paper, a direct ablation that swaps each verb+noun status phrase for a matched-length non-status paraphrase of the same step would separate the contribution of object-status semantics from the contribution of any shorter text anchor.","Beyond the paper, one could test the time-causal rule against a learned sequence model; if a learned order model matches or beats the hand-coded monotonicity rule, then the causal constraint is not the critical ingredient.","Beyond the paper, annotating frames where an ingredient is visibly whole versus visibly chopped and checking whether similarity to 'chopping carrots' tracks that visual change would distinguish true status recognition from text-image retrieval of a generic phrase."],"forward_implications":["If the reported gains are real, recipe progress can be tracked in real time from a single chest-mounted camera, without requiring the cook to hold a device or report each step by voice.","Because object-status phrases are tool-agnostic, the approach tolerates tool substitutions such as using a butter knife instead of a spatula, which the paper identifies as a common non-visual cooking practice.","The same status-anchoring idea should transfer to other multi-step hands-on activities where materials change state—makeup application, cleaning, crafting, or home repair—as the paper argues in its discussion.","The accuracy gap between curated videos and home sessions quantifies how much real-world conditions (lighting, camera framing, implicit tasks such as throwing away an eggshell) degrade current vision-language models.","The released 12-video dataset gives future accessibility-AI work a benchmark drawn from actual non-visual cooking practices rather than idealized studio footage."],"supporting_citations":[{"why":"Supplies the object-state-change idea and the [verb]+[noun] representation that OSCAR's status phrases are built from.","marker":"[54]"},{"why":"CLIP is one of the two vision-language models whose frame-step and frame-status similarity scores drive the predictions.","marker":"[42]"},{"why":"SigLIP is the second vision-language model, used to show the object-status gain is not specific to one encoder.","marker":"[55]"},{"why":"YouCook2 is the 173-video instructional dataset and supplies the annotation protocol reused for the real-world sessions.","marker":"[56]"},{"why":"Provides the cross-modal retrieval approach that motivates averaging similarity scores across frames to get a step prediction.","marker":"[34]"},{"why":"Supplies the temporal-causality constraint that OSCAR applies to keep step predictions monotonic over time.","marker":"[41]"},{"why":"GPT-4o is used to reformat raw recipe steps before object-status extraction.","marker":"[2]"}],"fun_headline_variants":["Object status phrases lift recipe-step accuracy by 20+ points","Object-state text boosts cooking step prediction by 20+ points","Cooking progress tracking gains from object status phrases","OSCAR: object status boosts step accuracy in cooking videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a text phrase pulled from the recipe, such as 'chopping carrots,' functions as visual recognition of the carrot's state, so if the vision-language models never actually track the transformation in the image, the reported gains are measuring better text-image retrieval rather than object-status recognition.","fun_headline_variants_meta":{"raw":{"variants":["Object status phrases lift recipe-step accuracy by 20+ points","Object-state text boosts cooking step prediction by 20+ points","Cooking progress tracking gains from object status phrases","OSCAR: object status boosts step accuracy in cooking videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3820,"prompt_tokens":966,"completion_tokens":2854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2787}},"tokens_in":582,"tokens_out":2854,"duration_ms":25576,"temperature":1.0,"reasoning_tokens":2787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:13:22.493787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 173-video evaluation with each object-status phrase replaced by a non-status paraphrase of equal length (for example, 'carrots in this step' in place of 'chopping carrots'); if accuracy stays at the OSCAR level, the gain is not caused by object status. Alternatively, take a set of frames in which the same ingredient is visibly whole and then visibly chopped and check whether CLIP/SigLIP similarity to the status phrase rises and falls with the actual visual state; if it does not, the pipeline is not recognizing object statuses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the object-state-change idea and the [verb]+[noun] representation that OSCAR's status phrases are built from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SigLIP is the second vision-language model, used to show the object-status gain is not specific to one encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YouCook2 is the 173-video instructional dataset and supplies the annotation protocol reused for the real-world sessions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the temporal-causality constraint that OSCAR applies to keep step predictions monotonic over time."}],"review_version":1}