{"id":"dce31ecd-8b63-4b14-b645-e220efed2472","arxiv_id":"2412.11026","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SceneLLM encodes video objects as learned discrete tokens, lets a LoRA-fine-tuned LLaMA reason over them, and decodes the hidden features into dynamic scene graph triplets, reporting state-of-the-art Recall@K on Action Genome.","lead":"This paper turns video frames into discrete token sequences that a language model can process, then decodes the model's hidden features into scene graph triplets describing object relationships. The authors report top scores on the Action Genome benchmark, but the paper omits code and leaves a key dimension mismatch unresolved.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The V2L-to-LLM pathway is not instantiable as written: VQ-VAE codebook vectors are 512-dimensional (Sec. 4.2) while LLaMA-13B token embeddings are 5120-dimensional, and no projection or alignment layer is described.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the VQ-VAE codebook dimension (512) is inconsistent with LLaMA-13B's token embedding dimension (5120), and no projection or alignment mechanism is described. This is not merely a missing implementation detail; it is an internal inconsistency between Section 3.2 and Section 4.2 that blocks the central computational pathway on which the SOTA claim and the LLM-reasoning conclusion depend. I agree with the reader's CONDITIONAL verdict: the issue is fixable in principle, but as written the architecture is not reproducible and the headline comparison cannot be verified. I also considered whether the causal attribution to the LLM is under-supported because the w/o LLM ablation removes parameters rather than isolating pre-trained knowledge, but the dimension mismatch is more fundamental and should be resolved first. No code is provided, so a dimension audit or released implementation is the minimal check that would settle whether the described system can run at all.","tokens_in":15737,"tokens_out":4102,"duration_ms":38483,"concrete_test":"Audit the tensor dimensions end-to-end: implement the pipeline exactly as described, with VQ-VAE latent and codebook dimension l = 512 and frozen LLaMA-13B (embedding/hidden dimension 5120), and run one forward pass on a single Action Genome clip. If codebook vectors are fed directly to LLaMA, the forward pass will fail with a shape mismatch unless an undocumented projection exists. Alternatively, inspect released code for a projection or embedding layer; if none exists, the described architecture is not reproducible. If a projection exists, report its dimensions and retrain or re-report results with it, because that projection is a substantive component absent from the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the claims of state-of-the-art results and of LLaMA's implicit knowledge driving the gain, the system must actually feed scene tokens into LLaMA. Section 3.2 states that the codebook dimension l \"is the same as the dimension of the word tokens in the LLM used,\" but Section 4.2 sets l = 512 and uses frozen LLaMA-13B. LLaMA-13B's token-embedding and hidden-state dimension is 5120, not 512. No projection or alignment layer is described anywhere, and Eq. (6) writes LLM(S_LLM) as if S_LLM were directly consumable by the LLM. The paper also does not specify how the implicit token sequence is merged with the text prompt \"Given such a scene sentence [S_LLM]...\"—whether as soft prompts, appended embeddings, or token IDs—nor how the autoregressive model M (a single convolutional GRU layer, Sec. 4.2) maps codebook outputs to LLaMA inputs. As written, the central V2L-to-LLM pathway cannot be instantiated, so the reported results cannot be checked against the described architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SceneLLM is a dynamic scene graph generation framework that first quantizes object-level ROI features with a VQ-VAE, then aggregates them into frame-level tokens via a Spatial Information Aggregation (SIA) scheme inspired by Chinese character composition, and finally uses Optimal Transport (OT) to form a video-level implicit linguistic signal. This signal is fed into a frozen LLaMA-13B fine-tuned with LoRA, and the LLM's final hidden features are decoded by a transformer-based SGG predictor into semantic triplets. The paper reports state-of-the-art results on Action Genome for PREDCLS, SGCLS, and SGDET under both constraint settings, along with ablations showing the contribution of the LLM, feature discretization, OT, and LoRA.","tokens_in":16018,"tokens_out":6534,"duration_ms":50165,"significance":"If the reported results were reproducible, the paper would be a useful demonstration that LLMs can serve as implicit scene analyzers for dynamic scene graph generation, an interesting and timely direction. The entropy derivation in Eq. (4) appears internally consistent, and the ablations (w/o LLM, w/ T5, w/o OT, w/o LoRA) are genuine comparative tests, not fitted quantities relabeled as predictions. However, the central empirical claims are undermined by a dimensional inconsistency in the LLM input pathway and by a state-of-the-art claim that Table 1 contradicts. The lack of code and variance reporting further limits verification. These issues must be resolved before the contribution can be accepted.","major_comments":[{"comment":"The manuscript claims in Section 3.2 that the VQ-VAE codebook dimension l is the same as the LLM's word-token dimension, but Section 4.2 sets l = 512 and uses frozen LLaMA-13B, whose token-embedding dimension is 5120. No projection or alignment layer is described between the codebook output and the LLM input, and Eq. (6) writes LLM(S_LLM) as if the scene tokens were directly consumable. Consequently, the central V2L-to-LLM pathway cannot be instantiated as described. The authors must either specify the projection/alignment mechanism, correct the dimension claim, or provide the actual implementation details that reconcile this inconsistency.","section":"Section 3.2 and Section 4.2"},{"comment":"The text claims that SceneLLM 'consistently achieves state-of-the-art results across all tasks and metrics.' This is contradicted by Table 1 in the With Constraint setting: for SGDET R@10, DDS achieves 36.2 while SceneLLM achieves 34.9. Thus the state-of-the-art claim is not supported for at least one metric. The authors should correct the claim and discuss the comparison with DDS, which is listed in the table but omitted from the comparison text.","section":"Section 4.3 and Table 1"},{"comment":"No code is provided, and all results in Tables 1, 2, and 3 are reported without variance or the number of seeds. As an empirical systems paper whose central claim is that a specific novel architecture achieves state-of-the-art performance, the absence of reproducibility details is a significant omission. At a minimum, the authors should report mean and standard deviation over multiple runs and make the code available.","section":"Section 4.2 and Tables 1-3"}],"minor_comments":[{"comment":"The abstract contains a typo: 'UA Vs' should be 'UAVs'.","section":"Abstract"},{"comment":"Algorithm 1 has a typo ('Flag=Flase') and the termination criterion is not guaranteed to halt; specify a maximum iteration or another early-stopping condition.","section":"Algorithm 1"},{"comment":"Eq. (2) does not specify the output dimension of the MLP that embeds the position information; please clarify how this dimension interacts with the codebook dimension.","section":"Eq. (2)"},{"comment":"The phrase in Section 3.2 that l is the same as the LLM word-token dimension conflicts with the implementation detail in Section 4.2; this must be reconciled even after a projection layer is added.","section":"Section 3.2 and Section 4.2"},{"comment":"Figures 2-4 would benefit from more detailed labels and captions; for example, the OT block in Fig. 4 is not described in the caption.","section":"Figures 2-4"},{"comment":"The claim of being 'the first to consider LLM as a scene analyzer through implicit language reasoning' should be supported by a more thorough discussion of recent LLM-based scene understanding systems.","section":"Contribution claim"}],"recommendation":"major_revision","confidential_remarks":"The dimensional inconsistency around the LLM input is the most serious issue and must be addressed with a concrete architectural description. The state-of-the-art claim contradicted by Table 1 also requires correction. If the authors cannot provide code or a detailed dimension schedule, the results cannot be independently verified, and the paper may need to be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth knowing about, but not as it stands. What is actually new: applying an LLM-as-implicit-reasoner to dynamic scene graph generation, with a V2L mapping that combines VQ-VAE discretization, a spatial aggregation module (hierarchical clustering + GCN), and OT-based codebook update to build a video-level token sequence. The OT entropy derivation in Eq. (4) is internally consistent, and the ablations (w/o LLM, w/ T5, w/o OT, w/o LoRA, w/o discretization) are genuine comparative tests, not fitted quantities relabeled as predictions. That is real evidence and deserves credit.\n\nThe soft spot is load-bearing. The stress-test note is right: Section 3.2 says the codebook dimension l matches the LLM word-token dimension, but Section 4.2 sets l=512 and uses frozen LLaMA-13B, whose token-embedding/hidden dimension is 5120. No projection or alignment layer is described anywhere, and Eq. (6) feeds S_LLM directly into the LLM. The paper also never specifies how the implicit token sequence merges with the prompt text. As written, the central V2L-to-LLM pathway cannot be instantiated, so the reported results cannot be reproduced from the architecture described. This is not a minor typo; it is a hole in the main claimed mechanism.\n\nTwo smaller issues. First, no code, seeds, or error bars are reported; Tables 1 and 2 are single runs, which is common in this subfield but limits confidence. Second, the ablation labeled \"w/o LoRA\" actually appears to be full fine-tuning (all parameters updated), which is a reasonable comparison but the name is misleading.\n\nIf the dimension mismatch is resolved—say, a projection layer was always there but omitted from the text, or the codebook is actually 5120-d—the paper could be a solid within-subfield contribution. As it stands, the central claim is unverifiable. Still, this deserves a serious referee: the task is useful, the recipe is plausible, and the flaw is fixable. I would not desk-reject it, but I would demand the authors clarify the projection and release code before believing the numbers.","headline":"SceneLLM reuses a known VQ-OT-LoRA recipe for a new task, but the central LLM-input pathway is internally inconsistent as written (512-d codebook vs 5120-d LLaMA-13B), so the SOTA claims cannot be checked.","tokens_in":16563,"tokens_out":1928,"would_cite":false,"duration_ms":17636,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SceneLLM claims that an LLM can act as a scene analyzer for dynamic scene graph generation by converting video frames into discrete 'scene token' sequences and decoding the LLM's hidden reasoning into semantic triplets, achieving…","keywords":["dynamic scene graph generation","large language model reasoning","video-to-language mapping","implicit linguistic signal","VQ-VAE discretization","optimal transport codebook","LoRA fine-tuning","Action Genome benchmark"],"falsifier":"Run the published SceneLLM code on a single video and inspect the tensor shapes at the boundary between the V2L module and LLaMA-13B: if the codebook vectors enter the model without passing through a projection that changes their dimension from 512 to 5120, the forward pass will fail and the claimed architecture is not instantiated. A second check is to replace LLaMA-13B with a randomly initialized transformer of the same size and measure SGCLS recall: if recall does not drop sharply, the LLM's pretrained knowledge is not doing the causal work attributed to it.","tokens_in":15517,"feed_emoji":"🧠","tokens_out":7575,"duration_ms":64446,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can serve as a scene analyzer for dynamic scene graph generation, not by asking the LLM to output text, but by turning video frames into a discrete token sequence that the LLM reasons over internally. The payoff would be a single framework that uses the LLM's pretrained world knowledge to infer subject-predicate-object triplets in video, improving recall over prior spatio-temporal models. On the Action Genome benchmark the authors report consistent state-of-the-art results across predicate classification, scene graph classification, and scene graph detection under both constraint settings. The ablations attribute the gain to the LLM itself, to discrete feature quantization, to the optimal-transport temporal codebook, and to LoRA fine-tuning. If the central claim is right, the next generation of video scene understanding can be built on implicit language reasoning rather than bespoke spatio-temporal architectures.","feed_headline":"SceneLLM beats prior video scene-graph models on Action Genome","feed_subtitle":"Frames become discrete 'scene sentences' that a fine-tuned LLM reasons over, then decodes into semantic triplets.","key_machinery":"The load-bearing mechanism is the Video-to-Language (V2L) mapping that turns continuous video features into discrete, language-like tokens. It has three parts: a VQ-VAE codebook whose vectors are meant to match the LLM's word-token dimension; a Spatial Information Aggregation (SIA) scheme, inspired by how Chinese characters combine radicals, that uses MLP position embedding, hierarchical clustering, and a GCN to produce one frame-level token per frame; and an optimal-transport update that merges frame-level tokens into an updated codebook with temporal semantics. These tokens, plus a short text prompt, are passed to a LoRA-fine-tuned LLaMA-13B, and the LLM's hidden states rather than its text output are decoded by a transformer SGG predictor. The OT codebook update and LoRA tuning are what let the pretrained LLM treat visual tokens as a 'scene sentence'.","core_discovery":"The central discovery claimed is that LLMs can reason about dynamic visual scenes when the video is converted into an implicit linguistic signal. SceneLLM's Video-to-Language mapping first quantizes object ROI features into codebook vectors with a VQ-VAE, aggregates them via a graph convolution based on hierarchical spatial clustering, and then uses optimal transport to merge frame-level tokens into a temporally informed codebook. The resulting token sequence is fed into a frozen LLaMA-13B fine-tuned with LoRA, and the LLM's final hidden features are decoded by a transformer-based SGG predictor into semantic triplets. The paper reports state-of-the-art recall on Action Genome and concludes that the LLM's implicit knowledge is what enables this, with ablations showing that removing the LLM, replacing it with T5, removing discretization, removing the OT scheme, or removing LoRA all degrade SGCLS performance.","pith_inferences":["Our inference: the same video-to-language interface could be applied to other video-understanding tasks, such as action recognition, video captioning, or open-vocabulary predicate prediction, by keeping the frozen LLM and swapping only the decoding head.","Our inference: the Chinese-character-inspired SIA gives each frame a compositional, graph-structured token, so probing the LLM's hidden states might reveal which object 'radicals' drive each predicted predicate, making the reasoning more inspectable than monolithic video features.","Our caution: because the stated codebook dimension (512) does not match LLaMA-13B's token-embedding dimension (5120), reproducing the method as written likely requires an unmentioned projection layer, and readers should check the released code before trusting the end-to-end pipeline."],"forward_implications":["Dynamic scene graph generation can be recast as an implicit language-reasoning problem, so advances in LLM architecture and pretraining transfer directly to video scene understanding.","Discrete visual tokens that mimic word tokens are a viable interface between vision encoders and LLMs, suggesting that explicit text or caption supervision is not required for LLM-based scene reasoning.","The optimal-transport codebook update provides a concrete way to compress frame-level spatial tokens into a temporally coherent sequence, which could replace hand-designed temporal modules in other video tasks.","Because SceneLLM reports state-of-the-art results on all three Action Genome tasks and settings, it sets a new numerical baseline that future dynamic SGG methods will need to beat on recall@K."],"supporting_citations":[{"why":"Supplies the VQ-VAE discrete quantization mechanism that converts continuous object features into codebook tokens.","marker":"[44]"},{"why":"Provides LLaMA-13B, the frozen large language model whose implicit knowledge and reasoning are the central claimed source of improvement.","marker":"[14]"},{"why":"Supplies the Low-Rank Adaptation method used to fine-tune the LLM on scene tokens while preserving its pretrained weights.","marker":"[25]"},{"why":"Provides the transformer-based SGG predictor D_SGG and the dynamic scene graph generation task formulation used for evaluation.","marker":"[8]"},{"why":"Defines the Action Genome benchmark and the PREDCLS, SGCLS, and SGDET evaluation protocol with recall@K metrics.","marker":"[26]"},{"why":"Supplies the autoregressive model M and the codebook-generation recipe used to produce the implicit linguistic signal.","marker":"[42]"},{"why":"Provides the optimal transport objective and Sinkhorn optimization used to build the temporally updated codebook.","marker":"[45]"},{"why":"Inspires the Spatial Information Aggregation scheme by representing frames as Chinese characters and objects as radicals.","marker":"[24]"},{"why":"Supplies the object detector used to extract ROI features and serves as a primary baseline for dynamic SGG.","marker":"[49]"}],"fun_headline_variants":["LLM reasons over video as implicit language for scene graphs","SceneLLM writes video as sentences for LLM graph reasoning","Optimal transport fuses frames into LLM-readable scene tokens","Video becomes LLM-implicit language, boosting scene graph recall","Frozen LLaMA fine-tuned to parse dynamic scenes via language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on feeding 512-dimensional codebook vectors into a model whose word-token embeddings are 5120-dimensional, with no described projection to bridge them; if that mismatch is real, the central scene-token-to-LLM pathway cannot run as written.","fun_headline_variants_meta":{"raw":{"variants":["LLM reasons over video as implicit language for scene graphs","SceneLLM writes video as sentences for LLM graph reasoning","Optimal transport fuses frames into LLM-readable scene tokens","Video becomes LLM-implicit language, boosting scene graph recall","Frozen LLaMA fine-tuned to parse dynamic scenes via language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000523,"raw_usage":{"total_tokens":2551,"prompt_tokens":993,"completion_tokens":1558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1471}},"tokens_in":609,"tokens_out":1558,"duration_ms":11108,"temperature":1.0,"reasoning_tokens":1471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:22:46.680070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published SceneLLM code on a single video and inspect the tensor shapes at the boundary between the V2L module and LLaMA-13B: if the codebook vectors enter the model without passing through a projection that changes their dimension from 512 to 5120, the forward pass will fail and the claimed architecture is not instantiated. A second check is to replace LLaMA-13B with a randomly initialized transformer of the same size and measure SGCLS recall: if recall does not drop sharply, the LLM's pretrained knowledge is not doing the causal work attributed to it.","supporting_citations":[{"cited_title":"Van Den Oord, O","cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE discrete quantization mechanism that converts continuous object features into codebook tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the transformer-based SGG predictor D_SGG and the dynamic scene graph generation task formulation used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Action Genome benchmark and the PREDCLS, SGCLS, and SGDET evaluation protocol with recall@K metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the autoregressive model M and the codebook-generation recipe used to produce the implicit linguistic signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the optimal transport objective and Sinkhorn optimization used to build the temporally updated codebook."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the Spatial Information Aggregation scheme by representing frames as Chinese characters and objects as radicals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the object detector used to extract ROI features and serves as a primary baseline for dynamic SGG."}],"review_version":1}