Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding object-status phrases to vision-language models lifts recipe-step prediction accuracy by 20–26 percentage points on both instructional videos and real non-visual cooking sessions.

desk verdict A useful dataset and a clear accessibility motivation, but the paper's central claim about object status is not isolated by the experiments—the gains could come from temporal smoothing or text ensembling. read the letter →

arxiv 2507.03330 v1 pith:ZMLJTBAN submitted 2025-07-04 cs.AI cs.CVcs.HC

Franklin Mingzhe Li , Kaitlyn Ng , Bin Zhu , Patrick Carrington This is my paper · ORCID
classification cs.AIcs.CVcs.HC
keywords objectstatusrecognitionrecipeprogresstrackingnon-visualcookingvision-languagemodelsCLIPSigLIPtime-causalmodelingaccessibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the evolving condition of ingredients and tools—an onion being chopped, a sauce thickening—is a usable signal for deciding which recipe step a cook is on, and that adding this signal to vision-language models makes step prediction much more accurate. The authors build OSCAR, a pipeline that parses recipes, extracts object-status phrases as verb-plus-noun pairs such as 'chopping carrots,' scores video frames against both recipe steps and these phrases with CLIP or SigLIP, and applies a monotonic time-causal rule so predictions cannot jump backwards. On 173 YouCook2 instructional videos, per-step accuracy rises from 41.7% to 68.0% with CLIP and from 62.2% to 82.8% with SigLIP; on 12 home cooking sessions recorded by blind and low-vision participants, it rises from 33.7% to 58.4% and from 41.9% to 66.7%. A sympathetic reader would care because this is a step toward assistive cooking tools that know what stage of a recipe a user is at without requiring the user to hold a phone or follow a rigid script.

What carries the argument

The load-bearing mechanism is the object-status phrase itself: a [verb]+[noun] description such as 'chopping carrots' that is extracted from the recipe text and used as an additional text anchor for video frames. A vision-language encoder (CLIP or SigLIP) produces a similarity score between each frame and each recipe step, and a second similarity score between the same frame and each step's object-status phrase; the two scores are averaged, and a time-causal model then suppresses predictions of an earlier step once a later step has been reached. The object status is never detected in the image by a dedicated model; it enters the pipeline as text, so the work the phrase does is entirely through text-image similarity.

What would settle it

Run the same 173-video evaluation with each object-status phrase replaced by a non-status paraphrase of equal length (for example, 'carrots in this step' in place of 'chopping carrots'); if accuracy stays at the OSCAR level, the gain is not caused by object status. Alternatively, take a set of frames in which the same ingredient is visibly whole and then visibly chopped and check whether CLIP/SigLIP similarity to the status phrase rises and falls with the actual visual state; if it does not, the pipeline is not recognizing object statuses.

Watch

Extended reading notes

Core claim

The central claim is that object status recognition consistently improves recipe-step prediction accuracy across vision-language models, in both curated instructional videos and real-world non-visual cooking. OSCAR represents object status as verb-plus-noun phrases extracted from the recipe's ingredient list and steps, computes CLIP/SigLIP similarity between each video frame and both the full step text and these status phrases, averages the two similarity scores, and then enforces time-causal monotonicity on predicted step numbers. The paper reports that this combination lifts CLIP from 41.7% to 68.0% and SigLIP from 62.2% to 82.8% on YouCook2, and from 33.7% to 58.4% and 41.9% to 66.7% on its own 12-session non-visual cooking dataset, with gains in every one of the 12 real-world sessions. The authors interpret the mechanism as disambiguation: status phrases pull apart visually similar or cluttered frames, while the time-causal rule prevents duplicate or out-of-order steps from being predicted.

Load-bearing premise

The load-bearing premise is that a text phrase pulled from the recipe, such as 'chopping carrots,' functions as visual recognition of the carrot's state, so if the vision-language models never actually track the transformation in the image, the reported gains are measuring better text-image retrieval rather than object-status recognition.

Editorial extensions

If this is right

  • If the reported gains are real, recipe progress can be tracked in real time from a single chest-mounted camera, without requiring the cook to hold a device or report each step by voice.
  • Because object-status phrases are tool-agnostic, the approach tolerates tool substitutions such as using a butter knife instead of a spatula, which the paper identifies as a common non-visual cooking practice.
  • The same status-anchoring idea should transfer to other multi-step hands-on activities where materials change state—makeup application, cleaning, crafting, or home repair—as the paper argues in its discussion.
  • The accuracy gap between curated videos and home sessions quantifies how much real-world conditions (lighting, camera framing, implicit tasks such as throwing away an eggshell) degrade current vision-language models.
  • The released 12-video dataset gives future accessibility-AI work a benchmark drawn from actual non-visual cooking practices rather than idealized studio footage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a direct ablation that swaps each verb+noun status phrase for a matched-length non-status paraphrase of the same step would separate the contribution of object-status semantics from the contribution of any shorter text anchor.
  • Beyond the paper, one could test the time-causal rule against a learned sequence model; if a learned order model matches or beats the hand-coded monotonicity rule, then the causal constraint is not the critical ingredient.
  • Beyond the paper, annotating frames where an ingredient is visibly whole versus visibly chopped and checking whether similarity to 'chopping carrots' tracks that visual change would distinguish true status recognition from text-image retrieval of a generic phrase.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OSCAR, a pipeline for recipe-progress tracking that uses GPT-4o to reformat recipes and extract object-status phrases of the form [verb]+[noun] (Section 3.1), computes CLIP and SigLIP similarity between video frames and both the recipe-step text and the object-status phrases (Section 3.2), averages the two similarity scores, and applies a time-causal model that enforces monotonic step order (Section 3.4). The pipeline is evaluated on 173 YouCook2 videos and a newly collected dataset of 12 real-world kitchen sessions recorded by blind and low-vision cooks. The reported results show large accuracy gains over a recipe-step-only baseline: from 41.7% to 68.0% (CLIP) and 62.2% to 82.8% (SigLIP) on YouCook2, and from 33.7% to 58.4% and 41.9% to 66.7% on the non-visual cooking dataset (Tables 1 and 2). The paper also presents error analyses and design considerations around implicit tasks, lighting, camera placement, and pre-prepared ingredients.

Significance. If the central claim were supported, the paper would make a useful contribution to accessible cooking technology and to procedural understanding for vision-language models. The real-world dataset of 12 non-visual cooking sessions is a genuine and valuable resource, and the design considerations in Section 5.5 are grounded in concrete observations. The pipeline is clearly described and the qualitative error analysis is thoughtful. However, the headline assertion that 'object status consistently improves step prediction accuracy' is not identified by the reported experiments: the OSCAR condition changes several components at once, and the object-status phrases are generated from the same recipe text that defines the baseline, so the measured gains cannot be attributed to visual object-status recognition as currently framed. The paper's strengths are the dataset, the domain motivation, and the design insights; the technical attribution needs further experiments.

major comments (4)
  1. [§4.2.2, Tables 1–2, §4.4.2] The central claim is not identifiable from the reported comparison. The OSCAR condition applies three changes simultaneously: (1) similarity against object-status phrases, (2) averaging of those scores with the recipe-step similarity scores, and (3) a time-causal model enforcing monotonic step order. The baseline applies none of these, so the 20+ point gains in Tables 1 and 2 measure the combined pipeline, not the effect of object status. The paper's own analysis attributes part of the improvement to the time-causal model (§4.4.2 for duplicate actions and §4.4.3, where 'adding the time-causal model corrected the prediction'), but no condition runs recipe-step similarity plus the time-causal model without object-status phrases, so the size of that contribution is unknown. The limitation section (§7) does not acknowledge this attribution gap. Please add at least four conditions — baseline, baseline+time-causal, baseline+object-status phrases without time-causal, and full OSCAR — and report per-video paired comparisons.
  2. [§3.1–§3.2] The construct validity of 'object status recognition' is not established. The object statuses are GPT-4o-extracted verb+noun phrases obtained from the ingredient list and cooking steps of the same recipe, and OSCAR never detects object state changes in the video; it computes CLIP/SigLIP text-image similarity between frames and these phrases. A shorter, differently worded paraphrase of the same step can improve retrieval accuracy without any recognition of object status, so the measured gain may reflect a reformatting or ensembling effect rather than the visual signal the title and RQ1 claim. A concrete test would be to compare the status phrases against control phrase sets (e.g., generic action phrases or paraphrases not derived from status semantics) and to evaluate whether the predicted statuses agree with manually annotated visual state changes.
  3. [§4.3, §5.3, Tables 1–2] The statistical support is insufficient for the claim that object status 'consistently improves' accuracy. No confidence intervals, paired significance tests, or per-video results are reported; the standard deviations are across videos, and the three repeated samples per step are not independent replicates. In particular, the assertion in §5.3.2 that 'These gains were consistent across all 12 videos' is unsupported without a per-video table, especially given the small n=12 dataset and the high variance reported. Please report per-video accuracies for all videos in both datasets and use a paired test (e.g., Wilcoxon signed-rank) to support the consistency claim.
  4. [§3.2, §4.2.2] The averaging of baseline and object-status similarity scores is underspecified. The text states that 'OSCAR then averaged the similarity metrics' (§3.2) and that 'the results were then averaged with the baseline similarity scores' (§4.2.2), but neither the weight nor the averaging formula is given, and no sensitivity analysis is provided. If equal weighting is intended, the magnitude of the reported gain depends on that untested hyperparameter; if another weighting was used, it should be stated. The blur-filter threshold and the exact GPT-4o prompt are also unspecified, which limits reproducibility of the pipeline.
minor comments (6)
  1. [§4.5.4] There is a typo: 'uoloaded' should be 'uploaded'.
  2. [§4.2.1] 'Groundtruth' should be 'ground truth' for consistency with the rest of the paper.
  3. [Throughout] The model name is spelled inconsistently as both 'SigLIP' and 'SigLip'; please standardize.
  4. [References] Reference [29], the CHI Extended Abstracts version, expands OSCAR as 'Object Status and Contextual Awareness for Recipes,' which differs from the acronym expansion used in this manuscript ('Object Status Context Awareness for Recipes'); please align the naming.
  5. [§5.3.2] The claim that gains were consistent across all 12 videos would be much easier to verify if a supplementary table of per-video baseline and OSCAR accuracies were included.
  6. [§4.1.2] The description of ingredient extraction says information was sourced 'directly from the video content or the publisher's notes on YouTube'; please clarify whether this was done by the authors and how disagreements were resolved, since this affects reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Object status is derived from the recipe-step text used as ground truth, and the reported gains bundle unablated temporal smoothing, making the central attribution partially circular.

  1. self definitional [Section 3.1-3.2; Table 1]
    "OSCAR then extracts object status information of a recipe by analyzing the ingredient list in conjunction with the associated cooking actions [54]. The extracted object statuses are represented in a structured format, typically as a combination of [verb] + [noun], such as 'chopping carrots' or 'sautéing mushrooms' [54]."

    The independent variable called 'object status' is generated from the same recipe-step text that defines both the baseline retrieval queries and the ground-truth step labels. A frame's similarity to 'chopping carrots' and to the recipe step 'Chop the carrots' are two text-image retrieval scores for the same underlying label; no detector validates that an object-state change actually occurred in the video. Thus the step-prediction target is embedded in the input by construction, and calling the averaged retrieval score 'object status recognition' renames query reformulation rather than measuring an independent visual object-state signal.

  2. other [Section 4.2.2; Tables 1-2; Section 4.4.2]
    "We then utilized the time-causal model to further enhance the performance (Section 3.4). ... The results were then averaged with the baseline similarity scores to obtain the final prediction accuracy to understand the usefulness of the object statuses compared to only using the recipe step text as a prediction [34]."

    The OSCAR condition changes three variables at once: status-phrase scores, averaging of status scores with baseline recipe-step scores, and a monotonic time-causal filter. Tables 1 and 2 report only the combined effect, while Section 4.4.2 concedes that 'implementing a time-causal model also improved performance.' Without a control condition that applies the time-causal model to recipe-step text alone, the reported +20-point gains cannot be attributed specifically to object status; part of the prediction improvement is produced by the unablated temporal machinery bundled into the OSCAR condition.

full rationale

OSCAR is evaluated against external CLIP/SigLIP models on YouCook2 and a newly collected dataset, and no accuracy parameter is fitted to the test labels, so the pipeline is not circular in the parameter-fitting sense. However, the central claim that 'object status consistently improves step prediction accuracy' is partially circular by construction: the object-status queries are produced by re-parsing the same recipe-step text that defines the baseline query and the ground-truth step labels, so part of the measured gain may be a paraphrase/format effect rather than an independent visual object-state signal. In addition, the OSCAR condition bundles status scores with score averaging and a time-causal monotonic filter, and no ablation isolates the status contribution; the paper itself credits the time-causal model with part of the improvement. These two issues make the attribution of the reported gains to object status unidentifiable, though the underlying retrieval computation is transparent and reproducible. Score 6 reflects partial circularity by construction, not full equivalence of the derivation to its input.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim depends on annotation quality, VLM similarity as a proxy, and the assumption that recipe-derived status phrases are valid visual labels. No physically invented entities are introduced. The evaluation protocol introduces several hand-chosen parameters, but none are fitted to the accuracy outcome.

free parameters (5)
  • frames_per_segment = 5
    One random frame is sampled from each of five equal segments per recipe step (Section 4.2.1); the choice of five changes the averaging and affects the reported accuracy.
  • prediction_repetitions = 3
    Each step prediction is repeated three times and averaged (Section 4.2.1); no seed is reported, so the exact numbers are not reproducible.
  • similarity_averaging_weight = 0.5 (implicit)
    The final OSCAR score averages baseline recipe-step similarity and object-status similarity (Sections 4.2.2, 5.2.2); equal weighting is adopted without sensitivity analysis.
  • blur_filter_threshold = not specified
    A blur filter selects the least blurred frame adjacent to the random sample (Section 4.2.1); the filter parameters are not given.
  • gpt4o_recipe_prompt = not specified
    GPT-4o reformats recipe steps and is used for object status extraction (Section 3.1); the prompt and validation protocol are not reported.
assumptions (6)
  • domain assumption YouCook2 temporal step annotations are accurate ground truth.
    Used as labels in Section 4.2.1 without re-annotation or quality checks.
  • domain assumption Manual ingredient extraction for the 173 videos is complete and correct.
    Section 4.1.2 describes manual review of videos and YouTube notes; no inter-annotator agreement is reported.
  • domain assumption CLIP and SigLIP cosine similarities are valid proxies for recipe-step and object-status alignment.
    Section 3.2 uses these scores directly as prediction evidence without calibration or human validation.
  • domain assumption Cooking steps proceed in a sequential order that the time-causal monotonic rule can enforce without distorting the evaluation.
    Section 3.4 prevents backward step predictions; real non-visual cooking includes interleaved and repeated actions, as Sections 5.4 and 5.5 acknowledge.
  • domain assumption Researcher annotations of the 12 non-visual cooking sessions capture the participants' intended steps.
    Section 5.1 describes following the YouCook2 annotation protocol; no participant verification is reported.
  • domain assumption Object status phrases extracted from recipe text correspond to visually observable food transformations.
    Section 3.1 derives statuses from ingredients and actions; the paper does not validate that each phrase is visible in the frame the model must match.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking." pith.science (2026). https://pith.science/paper/ZMLJTBAN

@misc{pith2026250703330,
  author       = {Pith},
  title        = {Pith review of: Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMLJTBAN}},
  note         = {Machine review of arXiv:2507.03330}
}
read the original abstract

Cooking plays a vital role in everyday independence and well-being, yet remains challenging for people with vision impairments due to limited support for tracking progress and receiving contextual feedback. Object status - the condition or transformation of ingredients and tools - offers a promising but underexplored foundation for context-aware cooking support. In this paper, we present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking. OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking. We evaluate OSCAR on 173 instructional videos and a real-world dataset of 12 non-visual cooking sessions recorded by BLV individuals in their homes. Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions, such as implicit tasks, camera placement, and lighting. We contribute the pipeline of context-aware recipe progress tracking, an annotated real-world non-visual cooking dataset, and design insights to guide future context-aware assistive cooking systems.

Figures

Figures reproduced from arXiv: 2507.03330 by the authors.

Figure 1
Figure 1. Illustration of OSCAR. (A) Recipe Formatting and Object Status Extraction, (B) Visual Data Extraction and Recipe Step [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Image frame of Y19 that shows a top-down view with all ingredients of making the burrito that impact the prediction accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Similarity scores for predicting the frame from step 3 (Y19), it shows step 3 is correctly predicted by OSCAR, but not the [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Image frames of cooking video that shows an exocentric view of the cooking steps. Left: This figure shows one frame in Y28, in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Image frames of cooking video that show similar visual frames of different steps that caused misprediction. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Image frame of cooking video that shows misprediction because of the descriptions against the norm. This figure shows the [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Image frames of cooking video that shows cooking BBQ pork fried rice and the cooking steam blurred the camera’s view. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Thumbnail of the non-Visual cooking dataset of 12 videos by people with vision impairments. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: These three frames were captured for V4 during step 1: ‘Crack an egg and scramble it.’ The middle frame showed the data [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Illustration of objects are not captured in the center of the field of view. Left: V8 is showing ‘Cut the cucumber into pieces [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Illustration of multiple objects was stored before cooking. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos

    cs.HC 2025-07 conditional novelty 6.0 of 10

    AROMA pairs a blind cook's spoken descriptions of what they feel, smell, and taste with a wearable camera and a video recipe to answer questions and issue proactive alerts, and eight participants rated it usable despi...

Reference graph

Works this paper leans on

59 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Siri Talks at You

    Ali Abdolrahmani, Ravi Kuber, and Stacy M Branham. 2018. " Siri Talks at You" An Empirical Investigation of Voice-Activated Personal Assistant (VAPA) Usage by Individuals Who Are Blind. InProceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility . 249–258

    work page 2018

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Dustin Adams and Sri Kurniawan. 2014. A blind-friendly photography application for smartphones. ACM SIGACCESS Accessibility and Computing 108 (2014), 12–15

    work page 2014

  4. [4]

    Niran Adetoro. 2016. INFORMATION ACCESS FOR THE VISUALLY IMPAIRED IN THE DIGITAL AGE

    work page 2016

  5. [5]

    Allrecipe. [n. d.]. Allrecipes | Recipes, How-Tos, Videos and More. https://www.allrecipes.com/. (Accessed on 04/18/2023)

    work page 2023

  6. [6]

    Jeffrey P Bigham, Chandrika Jayant, Andrew Miller, Brandyn White, and Tom Yeh. 2010. VizWiz:: LocateIt-enabling blind people to locate objects in their environment. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops . IEEE, 65–72

    work page 2010

  7. [7]

    Marie Claire Bilyk, Jessica M Sontrop, Gwen E Chapman, Susan I Barr, and Linda Mamer. 2009. Food experiences and eating patterns of visually impaired and blind people. Canadian Journal of Dietetic practice and research 70, 1 (2009), 13–18

    work page 2009

  8. [8]

    Paul Blenkhorn. 1995. Producing a text-to-speech synthesizer for use by blind people. Extra-ordinary Human-Computer Interaction: Interfaces for Users with Disabilities, ADN Edwards (ed.), Cambridge University Press, New York (1995), 307–314

    work page 1995

Show all 59 references
  1. [9]

    Gary Bradski. 2000. The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer 25, 11 (2000), 120–123

    2000

  2. [10]

    Stacy M Branham and Antony Rishin Mukkath Roy. 2019. Reading between the guidelines: How commercial voice assistant guidelines hinder accessibility for blind users. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility . 446–458

    2019

  3. [11]

    Da Cao, Zhiwang Yu, Hanling Zhang, Jiansheng Fang, Liqiang Nie, and Qi Tian. 2019. Video-based cross-modal recipe retrieval. In Proceedings of the 27th ACM International Conference on Multimedia . 1685–1693

    2019

  4. [12]

    Lucia Donatelli, Theresa Schmidt, Debanjali Biswas, Arne Köhn, Fangzhou Zhai, and Alexander Koller. 2021. Aligning actions across recipe graphs. In Proceedings of the 2021 conference on empirical methods in natural language processing . 6930–6942

    2021

  5. [13]

    Leah Findlater and Lotus Zhang. 2020. Input accessibility: A large dataset and summary analysis of age, motor ability and input performance. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility . 1–6

    2020

  6. [14]

    Tao Gong, Kai Chen, Xinjiang Wang, Qi Chu, Feng Zhu, Dahua Lin, Nenghai Yu, and Huamin Feng. 2021. Temporal ROI align for video object recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 1442–1450

    2021

  7. [15]

    Anhong Guo, Xiang’Anthony’ Chen, Haoran Qi, Samuel White, Suman Ghosh, Chieko Asakawa, and Jeffrey P Bigham. 2016. Vizlens: A robust and interactive screen reader for interfaces in the real world. In Proceedings of the 29th Annual Symposium on User Interface Software and Techn...

    2016

  8. [16]

    Danna Gurari, Qing Li, Chi Lin, Yinan Zhao, Anhong Guo, Abigale Stangl, and Jeffrey P Bigham. 2019. Vizwiz-priv: A dataset for recognizing the presence and purpose of private visual information in images taken by blind people. In Proceedings of the IEEE/CVF Conference on Compu...

    2019

  9. [17]

    Mina Huh, Zihui Xue, Ujjaini Das, Kumar Ashutosh, Kristen Grauman, and Amy Pavel. [n. d.]. Vid2Coach: Transforming How-To Videos into Task Assistants. ([n. d.])

  10. [18]

    Chandrika Jayant, Hanjie Ji, Samuel White, and Jeffrey P Bigham. 2011. Supporting blind photography. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility . 203–210

    2011

  11. [19]

    Nabila Jones, Hannah Elizabeth Bartlett, and Richard Cooke. 2019. An analysis of the impact of visual impairment on activities of daily living and vision-related quality of life in a visually impaired adult population. British Journal of Visual Impairment 37, 1 (2019), 50–63

    2019

  12. [20]

    Mohammad Kianpisheh, Franklin Mingzhe Li, and Khai N Truong. 2019. Face recognition assistant for people with visual impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–24. Manuscript submitted to ACM 24 Li et al

    2019

  13. [21]

    Minyung Kim, Sooyoung Hwang, Kyoungmin Choi, Youkeun Oh, and Dokshin Lim. 2022. Vision-based cooking assistance system for visually impaired people. In International Conference on Human-Computer Interaction . Springer, 540–547

    2022

  14. [22]

    Yonghee Kim, Youngju Park, and Jeongil Choi. 2017. A study on the adoption of IoT smart home service: using Value-based Adoption Model. Total Quality Management & Business Excellence 28, 9-10 (2017), 1149–1165

    2017

  15. [23]

    A König and K Thongpull. 2015. Lab-on-Spoon–a 3-D integrated hand-held multi-sensor system for low-cost food quality, safety, and processing monitoring in assisted-living systems. Journal of Sensors and Sensor Systems 4, 1 (2015), 63–75

    2015

  16. [24]

    Eliza Kostyra, Sylwia Żakowska-Biemans, Katarzyna Śniegocka, and Anna Piotrowska. 2017. Food shopping, sensory determinants of food choice and meal preparation by visually impaired people. Obstacles and expectations in daily food experiences. Appetite 113 (2017), 14–22

    2017

  17. [25]

    Jinna Lei, Xiaofeng Ren, and Dieter Fox. 2012. Fine-grained kitchen activity recognition using rgb-d. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 208–211

    2012

  18. [26]

    Franklin Mingzhe Li, Jamie Dorst, Peter Cederberg, and Patrick Carrington. 2021. Non-visual cooking: exploring practices and challenges of meal preparation by people with visual impairments. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Acc...

    2021

  19. [27]

    Franklin Mingzhe Li, Michael Xieyang Liu, Shaun K Kane, and Patrick Carrington. 2024. A Contextual Inquiry of People with Vision Impairments in Cooking. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–14

    2024

  20. [28]

    Franklin Mingzhe Li, Michael Xieyang Liu, Yang Zhang, and Patrick Carrington. 2022. Freedom to Choose: Understanding Input Modality Preferences of People with Upper-body Motor Impairments for Activities of Daily Living. In Proceedings of the 24th International ACM SIGACCESS Co...

    2022

  21. [29]

    Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, and Patrick Carrington. 2025. OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . 1–6

    2025

  22. [30]

    It feels like taking a gamble

    Franklin Mingzhe Li, Franchesca Spektor, Meng Xia, Mina Huh, Peter Cederberg, Yuqi Gong, Kristen Shinohara, and Patrick Carrington. 2022. “It feels like taking a gamble”: Exploring perceptions, practices, and challenges of using makeup and cosmetics for people with visual impa...

    2022

  23. [31]

    Franklin Mingzhe Li, Ashley Wang, Patrick Carrington, and Shaun K Kane. 2024. A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking Instructions. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility . 1–15

    2024

  24. [32]

    Franklin Mingzhe Li, Lotus Zhang, Maryam Bandukda, Abigale Stangl, Kristen Shinohara, Leah Findlater, and Patrick Carrington. 2023. Understanding Visual Arts Experiences of Blind People. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–21

    2023

  25. [33]

    Angela S Lin, Sudha Rao, Asli Celikyilmaz, Elnaz Nouri, Chris Brockett, Debadeepta Dey, and Bill Dolan. 2020. A recipe for creating multimodal aligned datasets for sequential tasks. arXiv preprint arXiv:2005.09606 (2020)

    2020 arXiv

  26. [34]

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860 (2021)

    2021 arXiv

  27. [35]

    Javier Marın, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2021. Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images. IEEE Transactions on Pattern Analysis and Machine Intel...

    2021

  28. [36]

    Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. 2021. Orbit: A real-world few-shot dataset for teachable object recognition. In Proceedings of the IEEE/CVF Internationa...

    2021

  29. [37]

    Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hendricks, Karolina Stańczak, and Aishwarya Agrawal

  30. [38]

    Food Network. [n. d.]. Recipes, Dinners and Easy Meal Ideas | Food Network. https://www.foodnetwork.com/recipes. (Accessed on 04/18/2023)

    2023

  31. [39]

    Patrick Olivier, Guangyou Xu, Andrew Monk, and Jesse Hoey. 2009. Ambient kitchen: designing situated services using a high fidelity prototyping environment. In Proceedings of the 2nd international conference on pervasive technologies related to assistive environments . 1–7

    2009

  32. [40]

    Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause and effect . Basic books

    2018

  33. [41]

    Karthir Prabhakar, Sangmin Oh, Ping Wang, Gregory D Abowd, and James M Rehg. 2010. Temporal causality for the analysis of visual events. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . IEEE, 1967–1974

    2010

  34. [42]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  35. [43]

    André Rodrigues, Kyle Montague, Hugo Nicolau, and Tiago Guerreiro. 2015. Getting smartphones to talkback: Understanding the smartphone adoption process of blind users. In Proceedings of the 17th international acm sigaccess conference on computers & accessibility . 23–32

    2015

  36. [44]

    Baifeng Shi, Qi Dai, Judy Hoffman, Kate Saenko, Trevor Darrell, and Huijuan Xu. 2021. Temporal action detection with multi-level supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 8022–8032

    2021

  37. [45]

    Yayoi Shimomura, Ebba Thora Hvannberg, and Hjalmtyr Hafsteinsson. 2010. Accessibility of audio and tactile interfaces for young blind people performing everyday tasks. Universal Access in the Information Society 9 (2010), 297–310. Manuscript submitted to ACM Exploring Object S...

    2010

  38. [46]

    Simon Stephan, Ralf Mayrhofer, and Michael R Waldmann. 2020. Time and singular causation—a computational model. Cognitive Science 44, 7 (2020), e12871

    2020

  39. [47]

    Dan Tasse and Noah A Smith. 2008. SOUR CREAM: Toward semantic processing of recipes. Carnegie Mellon University, Pittsburgh, Tech. Rep. CMU-LTI-08-005 (2008)

    2008

  40. [48]

    ChunYuen Teng, Yu-Ru Lin, and Lada A. Adamic. 2011. Recipe recommendation using ingredient networks. CoRR abs/1111.3919 (2011). arXiv:1111.3919 http://arxiv.org/abs/1111.3919

    2011 arXiv

  41. [49]

    Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morrison, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann

  42. [50]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35 (2022), 10078–10093

    2022

  43. [51]

    Trieu Minh Vu and Riva Khanna. 2018. Application of artificial intelligence in smart kitchen. International Journal of Innovative Technology and Interdisciplinary Sciences 1, 1 (2018), 1–8

    2018

  44. [52]

    Sagar M Waghmare, Kimberly Wilber, Dave Hawkey, Xuan Yang, Matthew Wilson, Stephanie Debats, Cattalyya Nuengsigkapian, Astuti Sharma, Lars Pandikow, Huisheng Wang, et al. 2025. Sanpo: A scene understanding, accessibility and human navigation dataset. In 2025 IEEE/CVF Winter Co...

    2025

  45. [53]

    Ru Wang, Nihan Zhou, Tam Nguyen, Sanbrita Mondal, Bilge Mutlu, and Yuhang Zhao. 2023. Practices and Barriers of Cooking Training for Blind and Low Vision People. (2023)

    2023

  46. [54]

    Zihui Xue, Kumar Ashutosh, and Kristen Grauman. 2024. Learning object state changes in videos: An open-world perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18493–18503

    2024

  47. [55]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 11975–11986

    2023

  48. [56]

    Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018. Towards Automatic Learning of Procedures From Web Instructional Videos. In AAAI Conference on Artificial Intelligence. 7590–7598. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17344

    2018

  49. [57]

    Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. 2019. R2gan: Cross-modal recipe retrieval with generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11477–11486. Manuscript submitted to ACM

    2019

  50. [2021]

    In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility

    Disability-first dataset creation: Lessons from constructing a dataset for teachable object recognition with blind and low vision data collectors. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility . 1–12

  51. [2024]

    arXiv preprint arXiv:2407.10920 (2024)

    Benchmarking Vision Language Models for Cultural Understanding. arXiv preprint arXiv:2407.10920 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.