Computer Vision and Pattern Recognition · arXiv:2606.14777 v2 · submitted 10 Jun 2026
JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
The paper says
“Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-25 · no language-model review
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.93 · states limitation 0.11 · human participants 0.65 · controlled comparison 0.79 · randomized assignment 0.02 · states quantitative result 0.83 · compares with prior results 0.94 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Many moments in the real world do not wait for a user to ask. | background 0.99 | 0.05 | 0.04 | asserted 1.00 |
| A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. | background 1.00 | 0.05 | 0.03 | asserted 0.96 |
| Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. | gap 0.52 | 0.07 | 0.06 | asserted 1.00 |
| We argue for a different paradigm: a model that is present in the world like a person. | gap 0.27 | 0.10 | 0.40 | asserted 0.82 |
| It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. | method 0.65 | 0.20 | 0.34 | asserted 0.91 |
| To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. | contribution 1.00 | 0.92 | 0.06 | asserted 0.97 |
| First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. | contribution 1.00 | 0.93 | 0.06 | asserted 1.00 |
| The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. | contribution 0.55 | 0.87 | 0.19 | asserted 1.00 |
| We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. | contribution 0.70 | 0.89 | 0.09 | asserted 1.00 |
| Second, we release a complete, deployable system built around that model. | contribution 1.00 | 0.94 | 0.06 | asserted 1.00 |
| The system streams any ongoing video into the model, making it genuinely present in the world. | method 0.72 | 0.87 | 0.06 | asserted 1.00 |
| All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. | method 0.56 | 0.83 | 0.04 | asserted 1.00 |
| Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. | result 1.00 | 0.94 | 0.23 | asserted 1.00 |
| To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system. | contribution 0.70 | 0.75 | 0.22 | asserted 0.87 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-25: 1 request, 10,204 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 18 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 32 citations worldwide. The full citation record
- MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation cs.CV · Nov 2026
- Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds cs.CV · Nov 2026
- ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video cs.CV · Oct 2026
- DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding cs.CV · Oct 2026
- AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video cs.CV · Nov 2026
- FuGIBench: Evaluating Reliable Preparation in Streaming Video cs.CV · Nov 2026
- TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs cs.CV · Oct 2026
- DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams cs.AI · Oct 2026
- Memory as Weights: Internalizing Long-Term History for Streaming Videos cs.CV · Oct 2026
- From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios cs.LG · Oct 2026
- VideoJev: System One Model for Video Understanding and Interaction cs.CV · Oct 2026
- EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos cs.CV · Oct 2026
- FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams? cs.CV · Oct 2026
- InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems cs.CV · Oct 2026
- OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction cs.CV · Oct 2026
- APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants cs.CV · Sep 2026
- TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding cs.CL · Sep 2026
- What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation cs.RO · Oct 2026
How the literature uses it
4 arXiv papers cite this in 4 sentences
What it cites · 15 sentences without a confident label
Uses the method or tool of 6 papers
-
We tokenize the video stream with AdaCodec [ref] (Figure [ref]), a predictive visual code inspired by predictive coding and video codecs: instead of re-encoding every frame, it transmits only what prediction cannot explain.
AdaCodec: A Predictive Visual Code for Video MLLMs · JoyAI-VL 1.0 and Architecture · 1.00
-
Following our streaming-native training [ref] , we therefore weight the assistant tokens by their role.
Harnessing Streaming Video in the Wild · Training Recipe of VL-Interaction · 0.87
-
JoyAI-VL 1.0 is obtained through three stages, representation alignment, vision-language pre-training and post-train with On-Policy Distillation [ref] and RL [ref] .
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models · JoyAI-VL 1.0 and Architecture · 0.97
Cites as background 8 papers
-
A research line lets models process video as it arrives rather than after the fact, spanning streaming video LLMs, proactive response [ref] , real-time inference [ref] , and long-horizon video memory [ref] .
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions · Streaming Video Understanding · 0.97
-
A research line lets models process video as it arrives rather than after the fact, spanning streaming video LLMs, proactive response [ref] , real-time inference [ref] , and long-horizon video memory [ref] .
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction · Streaming Video Understanding · 0.97
-
A research line lets models process video as it arrives rather than after the fact, spanning streaming video LLMs, proactive response [ref] , real-time inference [ref] , and long-horizon video memory [ref] .
StreamingVLM: Real-Time Understanding for Infinite Video Streams · Streaming Video Understanding · 0.97
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment