Pith.

Computer Vision and Pattern Recognition · arXiv:2606.14777 v2 · submitted 10 Jun 2026

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang

show 12 more authors

Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang

The paper says

“Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin.”

from the abstract · picked out as the main claim by a classifier

Pith says

Pith has not reviewed this paper.

Get a Pith Review, $5

Reviews

Reviews by people

No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-25 · no language-model review

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.93 · states limitation 0.11 · human participants 0.65 · controlled comparison 0.79 · randomized assignment 0.02 · states quantitative result 0.83 · compares with prior results 0.94 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Many moments in the real world do not wait for a user to ask. background 0.99 0.05 0.04 asserted 1.00
A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. background 1.00 0.05 0.03 asserted 0.96
Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. gap 0.52 0.07 0.06 asserted 1.00
We argue for a different paradigm: a model that is present in the world like a person. gap 0.27 0.10 0.40 asserted 0.82
It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. method 0.65 0.20 0.34 asserted 0.91
To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. contribution 1.00 0.92 0.06 asserted 0.97
First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. contribution 1.00 0.93 0.06 asserted 1.00
The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. contribution 0.55 0.87 0.19 asserted 1.00
We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. contribution 0.70 0.89 0.09 asserted 1.00
Second, we release a complete, deployable system built around that model. contribution 1.00 0.94 0.06 asserted 1.00
The system streams any ongoing video into the model, making it genuinely present in the world. method 0.72 0.87 0.06 asserted 1.00
All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. method 0.56 0.83 0.04 asserted 1.00
Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. result 1.00 0.94 0.23 asserted 1.00
To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system. contribution 0.70 0.75 0.22 asserted 0.87

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-25: 1 request, 10,204 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 18 Pith papers

Reviewed papers on Pith that cite this one. OpenAlex reports about 32 citations worldwide. The full citation record

  1. MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation cs.CV · Nov 2026
  2. Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds cs.CV · Nov 2026
  3. ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video cs.CV · Oct 2026
  4. DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding cs.CV · Oct 2026
  5. AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video cs.CV · Nov 2026
  6. FuGIBench: Evaluating Reliable Preparation in Streaming Video cs.CV · Nov 2026
  7. TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs cs.CV · Oct 2026
  8. DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams cs.AI · Oct 2026
  9. Memory as Weights: Internalizing Long-Term History for Streaming Videos cs.CV · Oct 2026
  10. From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios cs.LG · Oct 2026
  11. VideoJev: System One Model for Video Understanding and Interaction cs.CV · Oct 2026
  12. EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos cs.CV · Oct 2026
  13. FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams? cs.CV · Oct 2026
  14. InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems cs.CV · Oct 2026
  15. OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction cs.CV · Oct 2026
  16. APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants cs.CV · Sep 2026
  17. TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding cs.CL · Sep 2026
  18. What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation cs.RO · Oct 2026
How the literature uses it

4 arXiv papers cite this in 4 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 15 sentences without a confident label

Uses the method or tool of 6 papers
Cites as background 8 papers

Challenges and discussion

Know something about this paper? Leave a comment.

No comments yet.

Sign in to comment